"Duplicate Signature" error when writing to stdout using concurrent.futures.ProcessPool executor #541
Comments
|
I can confirm that the error goes away when I make the following changes:
def just_print(_):
print("Listen!")
from concurrent.futures import ProcessPoolExecutor
import multiprocessing as mp
from external import just_print
executor = ProcessPoolExecutor(mp_context=mp.get_context("spawn"))
list(executor.map(just_print, [None] * 1000))The cell no longer prints anything, and I cannot reproduce the error. This makes sense, as when using |
|
Yes, I was about to suggest the |
|
Indeed this is a fix, but it makes using |
|
Yes, that's what I meant by extra work for initialization. On the other hand, you may end up with less memory consumption, if that was a concern. |
|
You can copy https://github.com/nteract/scrapbook/blob/master/scrapbook/utils.py#L46-L57 to determine if you're in a kernel context or not and set the argument JIT before you call ProcessPoolExecutor. That way you don't need to make separate functions and can detect the right settings for the context. |
|
Unfortunately concurrency controls inside concurrent processes break some black box boundaries (both in and out of python). Async also has to take special steps based on the parent execution context, and threading mixed with multiprocessing can just fail straight out with horrible low level errors. Not sure jupyter_client itself can do anything better here to solve it :/ |
|
@davidbrochart @MSeal thanks for the quick response! I understand that using "spawn" or "forkserver" mode would be a proximal fix for the problem, however it does not satisfy other constraints that I have. For my use-case we make use of the fact that the subprocesses are forked to allow us to do all the required setup directly within the notebook, rather than doing everything in separate modules and later importing those modules. Specifically, this means that it is possible to Unfortunately one of the design constraints that I am working with is that this has to be possible, even if it can in principle lead to complete insanity (e.g. if the functions make use of mutable global state). |
|
I just learned about the A couple of questions about this:
|
|
@jbweston did you ever come to a conclusion on how you handle this (even if it's something hacky)? I've a similar situation where switching to spawn would be difficult/expensive. |
|
Not sure if this helps, but You can call the function by |
We're histting a relatively high tutorial job failure rate caused by jupyter/jupyter_client#541 on certain notebooks we run in the tutorials job. This appears to be caused by multiprocessing usage inside the qiskit that print to stdout while running. To try and avoid this issue, this commit disables parallelism in qiskit by setting the env var to do that. The only concern is whether we have sufficient time budget to execute the notebooks in CI.
We're histting a relatively high tutorial job failure rate caused by jupyter/jupyter_client#541 on certain notebooks we run in the tutorials job. This appears to be caused by multiprocessing usage inside the qiskit that print to stdout while running. To try and avoid this issue, this commit disables parallelism in qiskit by setting the env var to do that. The only concern is whether we have sufficient time budget to execute the notebooks in CI.
|
I think this was introduced by #493, which optimised I think resetting the session ID (different from the session key, which has to stay the same), as @jbweston suggested, would work. But a simpler approach might be to include the process ID in the message ID. At least on my system, The race condition is probably because there's a timestamp in the message header, so you only get the same signature if the same data is sent twice at the same time (microsecond precision, or as close to that as the system actually gives us). |
|
+1 on including the process ID in the message ID. |
Ensures messages are unique after fork Closes jupytergh-541
|
I've had a go at that in #655. |
Steps to reproduce
Create a single-celled notebook with the following contents:
Running this cell will result in "Listen!" being output into the browser 1000 times.
Run the cell a few times (the error is non-deterministic) and keep an eye on the log from the
NotebookApp. You should eventually see a traceback similar to the following:Environment
OS: WSL2 running Ubuntu 18.04 (Also been seen on other flavours of Linux and not under WSL2)
jupyter --versionconda env exportFurther information
Possibly related to #498, as
ProcessPoolExecutorusesmultiprocessingunder the hood.I think the most likely explanation is that the
ProcessPoolExecutoris usingforkmode for starting the subprocesses, which means that the subprocesses are using the same session key as the parent, which means that the signatures will sometimes clash.The text was updated successfully, but these errors were encountered: