[Ray] Implement gc for ray task executor context #3061

zhongchun · 2022-05-23T06:18:48Z

What do these changes do?

Ray executor holds all the object refs of a stage which is unnecessary. This PR is to free the unused object refs as soon as possible to reduce the memory peak and object store stress.

This PR includes the following parts:

Implements Ray task executor context gc.
Optimize task progress implementation.
Add exception and error handling
Add an option subtask_monitor_interval for RayExecutionConfig which can be set as:

task:
  execution_config:
    backend: ray
    ray:
      subtask_monitor_interval: 1.0

Related issue number

#2893

Check code requirements

tests added / passed (if needed)
Ensure all linting tests pass, see here for how to run them

…-cancel

Co-authored-by: Liu Bao <[email protected]>

…into feat-ray-executor-cancel

…-cancel

mars/services/task/execution/ray/executor.py

qinxuye · 2022-05-25T07:19:05Z

Could you explain a bit about ray task executor context gc, I am not aware of what problem you are solving.

fyrestone · 2022-05-25T07:52:58Z

Could you explain a bit about ray task executor context gc, I am not aware of what problem you are solving.

Currently, Ray executor holds all the object refs of a stage. This PR is to free the unused object refs as soon as possible to reduce the memory peak. We will update the description of this PR.

…-context-gc

fyrestone · 2022-05-27T02:04:11Z

mars/services/task/execution/ray/executor.py

+        total = len(subtask_graph)
+        completed_subtasks = OrderedSet()
+
+        deletion_seq = []


We don't need the deletion_seq, a mock _task_context can be used for testing.

Thanks for your reminder. I'll fix it.

fyrestone · 2022-05-27T02:10:54Z

mars/deploy/oscar/tests/test_ray_dag.py

+
+@require_ray
+@pytest.mark.asyncio
+async def test_detail_context_gc():


This test case can be moved to mars/services/task/execution/ray/tests/test_ray_execution_backend.py because no deployment is needed.

fyrestone · 2022-05-27T02:14:08Z

mars/services/task/execution/ray/executor.py

+            may be deleted.
+            """
+            i = 0
+            garbaged_subtasks = set()


garbage is a noun, please rename it to gc_subtasks or collected_subtasks.

Thanks for your reminder.

fyrestone · 2022-05-27T02:19:55Z

mars/services/task/execution/ray/executor.py

+                    ):
+                        yield
+                    if pred in garbaged_subtasks:
+                        continue


Checking subtask is collected should be moved to the beginning of the loop for pred in subtask_graph.iter_predecessors(subtask).

Yes, thanks.

…-context-gc

fyrestone

LGTM

qinxuye

LGTM

(cherry picked from commit 505361b)

…3116) Co-authored-by: bermaker <[email protected]>

zhongchun and others added 22 commits May 11, 2022 14:35

Implement cancel method for ray task executor

8bc2f16

Merge remote-tracking branch 'upstream/master' into feat-ray-executor…

990f750

…-cancel

some tests

4006efe

Fix test

7e1d080

Remove some unused code

772e808

Format

edca2a5

Fix exception processing

de0b710

Update mars/services/task/execution/ray/executor.py

d5b3ad6

Co-authored-by: Liu Bao <[email protected]>

Add some comments

e24e869

Merge branch 'feat-ray-executor-cancel' of github.com:zhongchun/mars …

f98fba3

…into feat-ray-executor-cancel

Fix test coverage

379080a

Format log and optimize subtask graph cancel

30715d8

Merge remote-tracking branch 'upstream/master' into feat-ray-executor…

ed5a260

…-cancel

Merge master and fix test

32e3fb2

Fix tests

2aadec2

Merge remote-tracking branch 'upstream/master' into feat-ray-executor…

41d43d7

…-cancel

Draft

afa2bea

add more ref counts

3d8df6c

Simplify context gc

81720a0

Add context gc test

1e13eae

Remove some debug message

8a23dd2

Remove unused imports

f6385dd

zhongchun requested a review from a team as a code owner May 23, 2022 06:18

zhongchun mentioned this pull request May 23, 2022

[PROPOSAL] The Execution API #2893

Open

zhongchun mentioned this pull request May 23, 2022

[Ray] Implement cancel method on Ray task executor #3044

Merged

2 tasks

fyrestone reviewed May 23, 2022

View reviewed changes

mars/services/task/execution/ray/executor.py Outdated Show resolved Hide resolved

mars/services/task/execution/ray/executor.py Outdated Show resolved Hide resolved

zhongchun added 2 commits May 23, 2022 22:50

Simplify gc

713faee

Remove unused variables and make ray.wait to async

a57befa

qinxuye added the type: enhancement request label May 24, 2022

Refine GC logic

339761d

Refine code

bd78bd7

刘宝 and others added 4 commits May 25, 2022 16:29

Rename finish -> completed

e4ec320

Refine code

4e09c1a

Merge remote-tracking branch 'upstream/master' into feat-ray-executor…

1fe63d4

…-context-gc

Add more test

d0715b7

fyrestone reviewed May 27, 2022

View reviewed changes

zhongchun added 2 commits May 27, 2022 12:17

optimize test

0fca89c

Merge remote-tracking branch 'upstream/master' into feat-ray-executor…

ad6958b

…-context-gc

zhongchun force-pushed the feat-ray-executor-context-gc branch from e139f12 to 3f89bd0 Compare May 27, 2022 06:49

Lint and fix test

b6e7b69

zhongchun force-pushed the feat-ray-executor-context-gc branch from 3f89bd0 to b6e7b69 Compare May 27, 2022 09:13

qinxuye changed the title ~~[Ray] Implement ray task executor context gc~~ [Ray] Implement gc for ray task executor context May 27, 2022

fyrestone approved these changes May 30, 2022

View reviewed changes

qinxuye approved these changes May 30, 2022

View reviewed changes

qinxuye merged commit 505361b into mars-project:master May 30, 2022

qinxuye pushed a commit to qinxuye/mars that referenced this pull request Jun 6, 2022

[Ray] Implement gc for ray task executor context (mars-project#3061)

2d49585

(cherry picked from commit 505361b)

qinxuye mentioned this pull request Jun 6, 2022

[BACKPORT][Ray] Implement gc for ray task executor context (#3061) #3116

Merged

wjsi pushed a commit that referenced this pull request Jun 6, 2022

[BACKPORT][Ray] Implement gc for ray task executor context (#3061) (#…

1ee28dc

…3116) Co-authored-by: bermaker <[email protected]>

wjsi added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels Jun 6, 2022

This was referenced Jun 6, 2022

[Ray] Fix Ray context GC #3118

Merged

[BUG] Ray context GC bug #3119

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ray] Implement gc for ray task executor context #3061

[Ray] Implement gc for ray task executor context #3061

zhongchun commented May 23, 2022 •

edited by fyrestone

Loading

qinxuye commented May 25, 2022

fyrestone commented May 25, 2022

fyrestone May 27, 2022

zhongchun May 27, 2022

fyrestone May 27, 2022

zhongchun May 27, 2022

fyrestone May 27, 2022

zhongchun May 27, 2022

fyrestone May 27, 2022 •

edited

Loading

zhongchun May 27, 2022

fyrestone left a comment

qinxuye left a comment

[Ray] Implement gc for ray task executor context #3061

[Ray] Implement gc for ray task executor context #3061

Conversation

zhongchun commented May 23, 2022 • edited by fyrestone Loading

What do these changes do?

Related issue number

Check code requirements

qinxuye commented May 25, 2022

fyrestone commented May 25, 2022

fyrestone May 27, 2022

Choose a reason for hiding this comment

zhongchun May 27, 2022

Choose a reason for hiding this comment

fyrestone May 27, 2022

Choose a reason for hiding this comment

zhongchun May 27, 2022

Choose a reason for hiding this comment

fyrestone May 27, 2022

Choose a reason for hiding this comment

zhongchun May 27, 2022

Choose a reason for hiding this comment

fyrestone May 27, 2022 • edited Loading

Choose a reason for hiding this comment

zhongchun May 27, 2022

Choose a reason for hiding this comment

fyrestone left a comment

Choose a reason for hiding this comment

qinxuye left a comment

Choose a reason for hiding this comment

zhongchun commented May 23, 2022 •

edited by fyrestone

Loading

fyrestone May 27, 2022 •

edited

Loading