Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(rust): Improve new-streaming groupby performance for high cardinality #19537

Merged
merged 3 commits into from
Oct 31, 2024
Merged

refactor(rust): Improve new-streaming groupby performance for high cardinality #19537

merged 3 commits into from
Oct 31, 2024
+568 −214

Conversation

orlp
Copy link
Collaborator

@orlp orlp commented Oct 30, 2024 •
edited
Loading

This is better, but still not good enough. We should do a local in-cache table for hot keys and have everything else spill into partitions instead of only partitioning once at the end.

@github-actions github-actions bot added internal An internal refactor or improvement rust Related to Rust Polars labels Oct 30, 2024
@orlp orlp marked this pull request as ready for review October 31, 2024 10:27
@ritchie46 ritchie46 merged commit bda606e into pola-rs:main Oct 31, 2024
21 of 22 checks passed
@c-peters c-peters added the accepted Ready for implementation label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 Awaiting requested review from ritchie46 ritchie46 is a code owner

@c-peters c-peters Awaiting requested review from c-peters c-peters is a code owner

Assignees

@orlp orlp

Labels
accepted Ready for implementation internal An internal refactor or improvement rust Related to Rust Polars
Projects
Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@orlp @ritchie46 @c-peters