Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] clean up ray scheduler threads after computing partial results #1597

Merged
merged 2 commits into from
Nov 13, 2023

Conversation

samster25
Copy link
Member

@samster25 samster25 commented Nov 13, 2023

@github-actions github-actions bot added the bug Something isn't working label Nov 13, 2023
Copy link

codecov bot commented Nov 13, 2023

Codecov Report

Merging #1597 (d70c3dd) into main (ef4d2fd) will decrease coverage by 0.27%.
Report is 2 commits behind head on main.
The diff coverage is 75.75%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1597      +/-   ##
==========================================
- Coverage   85.21%   84.94%   -0.27%     
==========================================
  Files          54       54              
  Lines        5180     5207      +27     
==========================================
+ Hits         4414     4423       +9     
- Misses        766      784      +18     
Files Coverage Δ
daft/runners/ray_runner.py 89.79% <75.75%> (-0.85%) ⬇️

... and 9 files with indirect coverage changes

@samster25 samster25 requested a review from jaychia November 13, 2023 18:56
@samster25 samster25 changed the title [BUG] clean up scheduler after computing partial results [BUG] clean up ray scheduler threads after computing partial results Nov 13, 2023
@@ -619,6 +634,12 @@ def __init__(
max_task_backlog=max_task_backlog,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to call stop_plan here after the while loops finish to clean up?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stop plan is called by the main thread to make this loop (in the background thread) complete and could be a race condition since the results may not be collected yet.

@samster25 samster25 merged commit 0052c17 into main Nov 13, 2023
42 of 43 checks passed
@samster25 samster25 deleted the sammy/clean-up-thread-for-partition-results branch November 13, 2023 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ray deadlocks on shutdown after Daft execution
2 participants