Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Who doesn't like multiprocessing? #84

Open
goodboy opened this issue Oct 18, 2019 · 3 comments
Open

Who doesn't like multiprocessing? #84

goodboy opened this issue Oct 18, 2019 · 3 comments
Labels
discussion enhancement New feature or request help wanted Extra attention is needed

Comments

@goodboy
Copy link
Owner

goodboy commented Oct 18, 2019

I've been hearing grumblings about distaste for multiprocessing. I can't discount these opinions because my inability to get a reliable system with multiprocessing is one of the main reasons I started building this project.

I do think it would be interesting to start some work on alternatives for process spawning/forking.

A couple ideas I have offhand:

Very open to opinions and ideas here!

@goodboy goodboy added discussion enhancement New feature or request help wanted Extra attention is needed labels Oct 18, 2019
@catern
Copy link

catern commented Oct 18, 2019

sfork is fairly experimental (and I never released a version of supervise that uses it - all the released versions use os.fork). rsyscall is the more robust form of it.

@goodboy
Copy link
Owner Author

goodboy commented Oct 18, 2019

@catern thanks for the clarification 👍 and my bad for the misread.

@goodboy
Copy link
Owner Author

goodboy commented Oct 21, 2019

Thanks to @ryanhiebert for finding trio_run_in_process which already implements the

use trio's new built-in process spawning as an alternative spawn method

approach!

goodboy added a commit that referenced this issue Apr 29, 2021
It's clear now that special attention is needed to handle the case where
a spawned `multiprocessing` proc is started but then the parent is
cancelled before the child can connect back; in this case we need to be
sure to kill the near-zombie child asap. This may end up being the
solution to other resiliency issues seen around mp with nested process
trees too. More testing is needed to be sure.

Relates to #84 #89 #134 #146
goodboy added a commit that referenced this issue May 6, 2021
It's clear now that special attention is needed to handle the case where
a spawned `multiprocessing` proc is started but then the parent is
cancelled before the child can connect back; in this case we need to be
sure to kill the near-zombie child asap. This may end up being the
solution to other resiliency issues seen around mp with nested process
trees too. More testing is needed to be sure.

Relates to #84 #89 #134 #146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants