Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ray] Improve Ray executor GC #3287

Merged
merged 23 commits into from
Nov 10, 2022
Merged

[Ray] Improve Ray executor GC #3287

merged 23 commits into from
Nov 10, 2022

Conversation

fyrestone
Copy link
Contributor

@fyrestone fyrestone commented Oct 31, 2022 •
edited
Loading

What do these changes do?

Ray executor submits subtasks do not suspend current aio task even await for loading subtask inputs. This could cause the GC is delayed to the submitting finish. This PR makes the executor switches the event loop to the GC aio task when submitting subtasks, reduces the load of object store.

  • Force switch aio task when Ray executior is submitting subtasks.
  • Cancel the monitor aio task when a stage is complete.
  • Clean the task context every stage complete, only reserves context that belongs to the task's result.
  • GC shuffle.
  • Support GC method: submitted and completed.
  • Init metrics for Ray executor.
  • Refine logs.
  • Pin xgboost < 1.7

Related issue number

Fixes #3216, Fixes #3290

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@fyrestone fyrestone self-assigned this Oct 31, 2022
@fyrestone fyrestone marked this pull request as ready for review November 2, 2022 06:03
@fyrestone fyrestone requested a review from a team as a code owner November 2, 2022 06:03
@fyrestone fyrestone marked this pull request as draft November 2, 2022 06:53
@fyrestone fyrestone changed the title [Ray] Force switch aio task when Ray executior is submitting subtasks [Ray] Improve Ray executor GC Nov 4, 2022
@fyrestone fyrestone added the type: enhancement request label Nov 4, 2022
@fyrestone fyrestone marked this pull request as ready for review November 7, 2022 10:44
zhongchun
zhongchun approved these changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@zhongchun zhongchun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

chaokunyang
chaokunyang approved these changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@chaokunyang chaokunyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chaokunyang chaokunyang merged commit 5a7abc6 into mars-project:master Nov 10, 2022
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Dec 9, 2022
@fyrestone @qianduoduo0904
[Ray] Improve Ray executor GC (mars-project#3287)
0a8e72a
aresnow1 pushed a commit to aresnow1/mars that referenced this pull request Dec 23, 2022
@fyrestone @aresnow1
[Ray] Improve Ray executor GC (mars-project#3287)
64f3656
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhongchun zhongchun zhongchun approved these changes

@chaokunyang chaokunyang chaokunyang approved these changes

Assignees

@fyrestone fyrestone

Labels
mod: ray integration mod: task service type: enhancement request
Projects
None yet
Milestone
No milestone
3 participants
@fyrestone @zhongchun @chaokunyang