Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Combine tree and shuffle methods in DataFrameGroupBy.agg tile #3051

Merged
merged 2 commits into from
May 23, 2022
Merged

Combine tree and shuffle methods in DataFrameGroupBy.agg tile #3051

merged 2 commits into from
May 23, 2022

Conversation

hekaisheng
Copy link
Contributor

@hekaisheng hekaisheng commented May 18, 2022 •
edited
Loading

What do these changes do?

This PR applies both tree and shuffle methods for DataFrameGroupBy.agg's auto method. First, combine map chunks into larger chunks according to sample chunk's size, then shuffle those combined chunks. It brings improvements when executing groupby on cluster.

Related issue number

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@hekaisheng hekaisheng added type: enhancement request mod: dataframe to be backported Indicate that the PR need to be backported to stable branch labels May 18, 2022
@hekaisheng hekaisheng added this to the v0.10.0a1 milestone May 18, 2022
@hekaisheng hekaisheng marked this pull request as ready for review May 19, 2022 10:21
@hekaisheng hekaisheng requested a review from a team as a code owner May 19, 2022 10:21
qinxuye
qinxuye previously approved these changes May 19, 2022
View reviewed changes
Copy link
Collaborator

@qinxuye qinxuye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hekaisheng @wjsi
Refine
3bfb750 
Co-authored-by: Wenjun Si <[email protected]>
wjsi
wjsi approved these changes May 23, 2022
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

qinxuye
qinxuye approved these changes May 23, 2022
View reviewed changes
Copy link
Collaborator

@qinxuye qinxuye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qinxuye qinxuye merged commit d2fde16 into mars-project:master May 23, 2022
@qinxuye qinxuye deleted the enh/groupby-agg branch May 23, 2022 07:42
hekaisheng added a commit to hekaisheng/mars that referenced this pull request May 24, 2022
@hekaisheng
Combine tree and shuffle methods in DataFrameGroupBy.agg tile (mars…
12f4f26 
…-project#3051)

(cherry picked from commit d2fde16)
@hekaisheng hekaisheng mentioned this pull request May 24, 2022
Merged
wjsi pushed a commit that referenced this pull request May 24, 2022
@hekaisheng
[BACKPORT] Combine tree and shuffle methods in DataFrameGroupBy.agg…
347688c 
… tile (#3051) (#3077)
@wjsi wjsi added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels May 24, 2022
@hekaisheng hekaisheng mentioned this pull request May 25, 2022
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qinxuye qinxuye qinxuye approved these changes

@wjsi wjsi wjsi approved these changes

Assignees
No one assigned
Labels
backported already PR has been backported mod: dataframe type: enhancement request
Projects
None yet
Milestone
v0.10.0a1
Development

Successfully merging this pull request may close these issues.
3 participants
@hekaisheng @qinxuye @wjsi