After update 'zmq.error.ZMQError: Operation not permitted' problem

[ManPython]

Describe the bug
I can't recognize the real source of problem as it is IPython problem or notebook, cython, zmq, ipykernel ?
ipython/ipython#13867

[I 00:09:11.918 NotebookApp] KernelRestarter: restarting kernel (4/5), new random ports
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/ipykernel_launcher.py", line 17, in <module>
    app.launch_new_instance()
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/traitlets/config/application.py", line 1040, in launch_instance
    app.initialize(argv)
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/traitlets/config/application.py", line 113, in inner
    return method(app, *args, **kwargs)
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 666, in initialize
    self.init_sockets()
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 307, in init_sockets
    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 244, in _bind_socket
    return self._try_bind_socket(s, port)
  File "/usr/home/manpython/venvp311/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 220, in _try_bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 540, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Operation not permitted
[W 00:09:14.941 NotebookApp] KernelRestarter: restart failed

• OS: [e.g. iOS]
  NAME=FreeBSD
  VERSION=13.0-RELEASE-p11
  VERSION_ID=13.0

• Browser [FF]

• Version [108.0.1]

Additional context
I upgraded everyting what was possible around jupyter. Works fine before I decided move to P311, no sucess then back to P378.. and ops.. this same error.

[RRosio]

Hi @ManPython thank you for submitting this issue. Is this a problem that you also face when running Jupyter Lab? This would be helpful in identifying whether this is a Jupyter issue or not.

[ManPython]

Hi @ManPython thank you for submitting this issue. Is this a problem that you also face when running Jupyter Lab? This would be helpful in identifying whether this is a Jupyter issue or not.

Yes, pure jupyter and jupyterlab.

[ManPython]

Looks it's can be problem due fact that jupyter not respecting the parameters for tcp as:

# KernelManager(ConnectionFileMixin) configuration
c.KernelManager.connection_file = /somehting.json

And the ports, coz not taking into --debug vars from c.KernelManager.connection_file and the ports, etc. always random.
Then if OS require reservation for ports, and are dedicated and jupyter taking random free, then this problem.
https://groups.google.com/g/jupyter/c/qamkem52Xn0

[ManPython]

I made some test with downgrade around and I was suprised that no any version not respecting settings around c.KernelManager.connection_file or related c.KernelManager objects. Sounds like critical bug and issue around secuirty.

[RRosio]

Hi @ManPython thank you for the updates. A follow up question from triage, could you verify if any of the settings in the file where you are adding configuration details get loaded? We're also linking Jupyter_Client where the KernelManager reside, as it could be a matter of loading the settings to a different Kernel Manager class. (Thanks to @krassowski)

[ManPython]

The settings not working from json as file and as static config .py around KernelManager.
Soo.. it's depth issue around, not have time to fixe this all by own on this moment, the test taking many times, coz was not sure where is problem.
2 thinks.. the FreeBSD limitations in relation to hosting configuration for preserved ports and some uknow reason around IPython/Kernel.. in fact.. if can't force KernelManager ports to reserved, can't exclude issue around FreeBSD as first, if this can be pass with fix around, then if no solved problem, then IPython/ZMQ..

[kevin-bates]

Hi @ManPython - I'm not entirely sure I'm following the complete issue here, but regarding the c.KernelManager.connection_file configurable property, I believe there might be a misunderstanding and I'm hoping the following might clarify some things (at the risk of confusing them more 😉).

My understanding is that attribute is configurable in name only. That is, the user can provide what is essentially the name of where they want configuration information stored, but any existing contents are not used and, indeed, overwritten with the port information determined at runtime [*].

Also, because multi-kernel applications, like Jupyter Notebook or Lab, can create and manage multiple kernels, the configurable setting is (at least up to today) ignored (and is where the kernel-<kernel_id>.json format comes from). As a result, the c.KernelManager.connection_file configuration (in name only) is really meant for single-kernel applications like those built on nbclient, etc.

[*] Another configurable approach that could be taken (and, again, really only applies to single-kernel applications) is that each of the ZMQ port values can be configured. As a result, I believe one could set their port values, then execute a notebook using nbclient with config values like c.KernelManager.shell_port = 32123, c.KernelManager.control_port = 32124, etc.

It sounds like you'd like the ability to participate in environments in which port ranges are restricted. If so, the only Jupyter application that I'm aware of that enables the ability to honor port ranges is Jupyter Enterprise Gateway. It can be configured to choose ports from within a given range, as can the kernels it spawns. Because it is primarily used for launching remote kernels, each kernelspec can specify its own range of ports that, presumably, apply to the configuration on which kernels of that specification will run.

[ManPython]

As a result, I believe one could set their port values, then execute a notebook using nbclient with config values like c.KernelManager.shell_port = 32123, c.KernelManager.control_port = 32124, etc.

And this is what not working for me, the ports are random due kernel-<kernel_id>.json with hash for each start/restart.
I tries prevent this random hashed pid by constant .json as c.KernelManager.connection_file or static vars in .py but no sucess here.

Because it is primarily used for launching remote kernels, each kernelspec can specify its own range of ports that, presumably, apply to the configuration on which kernels of that specification will run.

But exist option that FreeBSD and admin require to usage only preserved ports due fact many users can use own and can be conflict, by this in some hosting panels are option ot reserve ports, and if user ussing other not have acess. Anyway.. if can't confirm seted own static ports from vars, then hard to say where is issue if not here.

[kevin-bates]

As a result, I believe one could set their port values, then execute a notebook using nbclient with config values like c.KernelManager.shell_port = 32123, c.KernelManager.control_port = 32124, etc.

And this is what not working for me, the ports are random due kernel-<kernel_id>.json with hash for each start/restart.
I tries prevent this random hashed pid by constant .json as c.KernelManager.connection_file or static vars in .py but no sucess here.

If you're seeing a connection file name like kernel-<kernel_id>.json (since that filename format is introduced by the MultiKernelManager), then you're attempting to configure items that are not meant to be configured because it's a multi-kernel application. Did you try configuring the port values within the context of a single-kernel application like nbclient (and not jupyter-notebook or jupyterlab)?

But exist option that FreeBSD and admin require to usage only preserved ports due fact many users can use own and can be conflict, by this in some hosting panels are option ot reserve ports, and if user ussing other not have acess. Anyway.. if can't confirm seted own static ports from vars, then hard to say where is issue if not here.

The existing kernel framework does have an inherent race condition in play where the application creates the connection file, but the kernel (following its process launch) is what actually binds to the ports. As a result, those ports can be bound by other, related or unrelated, applications. This is a known issue within the ecosystem. EG mitigates this by asking a process that embeds the kernel to allocate the ports prior to starting the kernel (essentially eliminating the overhead of process startup and significantly decreasing the window of the race condition), but this can't be completely resolved until the kernel itself creates/binds the ports itself.

How frequently is this error occurring? When it does occur are you able to determine what kind of load/activity is taking place on the server? If this is due to port conflicts, then the issue should be relatively intermittent, although would likely happen more frequently on more heavily loaded servers.

[ManPython]

If you're seeing a connection file name like kernel-<kernel_id>.json (since that filename format is introduced by the MultiKernelManager), then you're attempting to configure items that are not meant to be configured because it's a multi-kernel application. Did you try configuring the port values within the context of a single-kernel application like nbclient (and not jupyter-notebook or jupyterlab)?

It was plan exactly to call a kernel-id_1.json coz each restart creating many files, for me is stupid, and then in .json address available and reserved ports.

1. By this is not clear exactly.. if new tabs with jupyter notebook or jupyter lab, then we have new kernel-<kernel_id>.json ? 2. And this same if "multiplykernel" like 1 tab/file ipynby python27, 2 tab/file ipynby python30, 3 tab/file ipynby python311? Can be for this one kernel-id_1.json and this same ports? Or due multiply tabs/files must be many kernel_id and many ports?

Did you try configuring the port values within the context of a single-kernel application like nbclient (and not jupyter-notebook or jupyterlab)?

This same error around title issue is for jupyter notebook and jupyter lab, coz connection problem/sockets. Jupyter lab don't have own configuration around, I think.. not sure.

I can say, that everything works time ago, but don't know what was changed and when.. eg. server update, configuration server update, or due my updated around jupyter, python, and adding 2 kernel as python311. And regarding venv I'm running jupyter by python37 and python311 and both problems are this same. But.. some accidental situations, where some kernels working, and code in jupyter as simple example works, but don't know reason why working sometimes as 1% of restarts.

[kevin-bates]

It was plan exactly to call a kernel-id_1.json coz each restart creating many files, for me is stupid, and then in .json address available and reserved ports.

The (up to) 5 restart attempts you're seeing...

[I 00:09:11.918 NotebookApp] KernelRestarter: restarting kernel (4/5), new random ports

take place when the IOLoopKernelManager (which is typically only used by multi-kernel applications) detects the kernel process no longer exists. In these "automatic restart" scenarios, both the kernel-id and ports do not change (which is unfortunate when the nature of the error is a port conflict) - so you should not be seeing, for example, 5 additional kernel-<kernel_id>.json files.

1. By this is not clear exactly.. if new tabs with jupyter notebook or jupyter lab, then we have new kernel-<kernel_id>.json ?

Correct. Each kernel instance, which is one-to-one with a "running" notebook, which is one-to-one with a tab, will be associated with a kernel-<kernel_id>.json file - so 5 tabs == 5 kernel-<kernel_id>.json files (each of which reference 5 different port and signature key values).

2. And this same if "multiplykernel" like 1 tab/file ipynby python27, 2 tab/file ipynby python30, 3 tab/file ipynby python311? Can be for this one kernel-id_1.json and this same ports? Or due multiply tabs/files must be many kernel_id and many ports?

Each kernel instance uses 5 ZMQ ports. So if you have 3 kernels active on a server, those kernels are consuming 15 ports. Kernel implementations consume a connection file that contains that kernel's port and signature key information and nothing related to another kernel.

. But.. some accidental situations, where some kernels working, and code in jupyter as simple example works, but don't know reason why working sometimes as 1% of restarts.

If your server is heavily loaded or has limited ports available for applications, this is the kind of behavior you'd see (relatively intermittent). However, if you're only seeing "simple" notebooks working, you may have issues with the more complex notebooks (and their associated kernels) running and exiting due to those kinds of issues. Those unexpected exits, however, would not occur until the first cell is submitted for execution. So, as a result, you would not see the sequence of 1 thru 5 restarts occurring because, in those cases, the kernel is able to start. So, you'd likely see automatic restarts succeed after the first attempt. Then, when the first cell (or problematic cell) is submitted, see the kernel restarted - indicating that that cell's content is causing the kernel's unexpected termination.

I would recommend emulating your same environment on another server or desktop in order to capture data points from other configurations and try to determine if this is due to that particular server's configuration (or activity).

[ManPython]

The (up to) 5 restart attempts you're seeing...

More as restart via terminal or server as hardware, etc. more as result experiments around configurations.

Correct. Each kernel instance, which is one-to-one with a "running" notebook, which is one-to-one with a tab, will be associated with a kernel-<kernel_id>.json file - so 5 tabs == 5 kernel-<kernel_id>.json files (each of which reference 5 different port and signature key values).

By this answer is clear that is no sense to set static values for ports and any attempts to set for future with reservation not have sense..
Only if planing have run 1 notebok, not close, maybe to usage via API, etc. or call.. not sure.
Soo. I was wrong with this idea that static can solve problem.. I experimented a lot as option to find source of issues.
Due this.. stil don't know why this error exist.

I would recommend emulating your same environment on another server or desktop in order to capture data points from other configurations and try to determine if this is due to that particular server's configuration (or activity).

I think this not have sense and it's not solution due on windows, desktop working, etc. but only FreeBSD I meet this issue. Eg. I have jupyter on p27, p37, p311 as venv and time ago works well, but after some updated around python (exactly when installed p311) all options not working normally.

[ManPython]

The issue is reagardin tcp "transport": "tcp", in case icp.