-
Notifications
You must be signed in to change notification settings - Fork 3.7k
enhance: map multi row groups into one cache cell #46249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: sparknack The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
[ci-v2-notice]
To rerun ci-v2 checks, comment with:
If you have any questions or requests, please contact @zhikunyao. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #46249 +/- ##
==========================================
- Coverage 75.93% 75.93% -0.01%
==========================================
Files 1898 1898
Lines 297301 297412 +111
==========================================
+ Hits 225757 225834 +77
- Misses 64047 64085 +38
+ Partials 7497 7493 -4
🚀 New features to boost your workflow:
|
2607bda to
667c490
Compare
Signed-off-by: Shawn Wang <[email protected]>
667c490 to
356b73e
Compare
issue: #45486
Introduce row group batching to reduce cache cell granularity and improve
memory&disk efficiency. Previously, each parquet row group mapped 1:1 to a cache
cell. Now, up to
kRowGroupsPerCell(4) row groups are merged into one cell.This reduces the number of cache cells (and associated overhead) by ~4x while
maintaining the same data granularity for loading.