Additional to the actual signal, send a message on the control port #294

filmor · 2017-09-22T14:48:11Z

In Erlang I have only on POSIX and only in the newest version some control over the normal runtime signals, in particular SIGINT. However, I can easily pick up and process the respective message on the control socket to interrupt execution.

Can this functionality be added? If yes, I'd continue and document the functionality and add tests. Maybe it would also be good to have the option of not sending an actual signal (via os.kill) at all.

SpencerPark · 2017-10-07T18:09:02Z

I was just looking for something similar to this, specifically for interrupting the kernel. Sending something over the control channel makes complete sense for this. It is a high priority message like the shutdown request. I don't see the benefit of sending the signals over sending a control message.

Since we are at a higher level when working at the messaging level I think that the message doesn't need to be signal related anymore but rather simply an interrupt_request or similar. What do you think?

rgbkrk · 2017-10-08T01:48:04Z

Cc @ivanov

takluyver · 2017-10-09T15:13:40Z

The rationale for signals is that the kernel may block message processing while it is executing user code, so a new message - even on the control channel - will not necessarily interrupt it. This is the case in the reference Python kernel, in the R kernel, and probably in all kernels built on the Python 'wrapper kernel' machinery, e.g. the bash kernel.

We have also thought of adding messages for signals in a different context, to allow interrupting kernels running remotely. In that proposal, the frontend would send a signal message to a 'kernel nanny', a process running alongside the kernel, which would respond by sending a real Unix signal to the kernel process.

IIRC it also came up there that some kernels would rather get an interrupt message than a signal, so it sounds like this might be worth doing.

I'm not entirely convinced that sending both the signal and the message is the right thing to do. I don't yet have a concrete reason why not, but it feels a bit messy. Maybe kernels should have a way to declare whether they expect signals as real signals or as messages?

SpencerPark · 2017-10-09T18:22:29Z

Fair enough, I think an opt-in for the interrupt message is a good compromise. This would be backwards compatible as well which is a big plus. A kernel.json flag seems like a good place for a kernel to make that request. I don't think it matters if the front-end acknowledges the request as currently kernels that don't handle the interrupt signal simply don't support interrupts. If a kernel requests an interrupt message as the method of communication and never receives one then the behavior is simply the same as it is currently.

My understanding of the control channel was that it should try to always be free to handle the high priority requests so when implementing the protocol I built with that in mind. It is on a separate thread and is therefore available to handle the request while the main shell is busy.

takluyver · 2017-10-10T10:40:29Z

The way we implement the control channel is that it can pre-empt queued messages on the shell channel. So if you send 10 cells for execution and then immediately ask the kernel to shut down, it will finish executing the first cell and shutdown before starting the other 9. But only signals pre-empt something that's already running.

filmor · 2017-10-10T12:43:13Z

Is this behaviour of the control socket that a requirement, though? In Erlang I can easily interrupt and shut down execution while a cell is evaluated.

I think I agree with scoping this down to an interrupt_request instead of trying to handle arbitrary signals. However, there should still be a way to deactivate any kind of signal-use (apart from TERM). Erlang handles the situation kind-of gracefully, I think, but this is a property of the runtime that I can't necessarily influence from the kernel implementation.

Is there any scenario in which sending the interrupt_request as a message on the control socket unconditionally could break a kernel?

takluyver · 2017-10-10T12:58:03Z

Is this behaviour of the control socket that a requirement, though?

Not particularly, or at least it's not really specified, as far as I know. I'm just explaining the background of why signals are needed. :-)

Is there any scenario in which sending the interrupt_request as a message on the control socket unconditionally could break a kernel?

I think it would be fine for a well-written kernel, but it could cause problems if a kernel author doesn't realise that they're getting both a signal and a message, and the kernel gets interrupted twice. I think it's common for kernels to print something like warning: unknown message type on unhandled messages, so if people start seeing unhandled interrupt_request messages, they're going to go and 'fix' that, even if interrupting already works with SIGINT.

Allowing the kernel to pick either messages or signals will hopefully avoid this kind of confusion.

filmor · 2017-10-18T22:31:38Z

Okay, I'll update the PR accordingly.

filmor · 2017-10-19T09:26:49Z

Updated, please have another look.

SpencerPark · 2017-10-19T16:43:06Z

jupyter_client/manager.py

+                if sys.platform == 'win32':
+                    from .win_interrupt import send_interrupt
+                    send_interrupt(self.kernel.win32_interrupt_event)
+                    self._send_signal_message(signal.SIGINT)


I don't think the _send_signal_message applies anymore. Might be left over from the initial commit?

You're right, fixed :)

takluyver · 2017-10-20T09:26:26Z

Thanks. Since this involves a change to the message protocol, I'm going to start a discussion on the mailing list.

lresende · 2017-10-23T02:37:26Z

@kevin-bates I want to make sure you see this, as this might be similar to what you did for remote kernel interrupt in "Enterprise Gateway"

kevin-bates · 2017-10-23T14:58:44Z

Thanks for the heads up @lresende. Yes, I had seen this a few days ago.

The kernel launchers we use in Enterprise Gateway appear to follow the kernel nanny model described above, although our launchers actually house the target kernel.

Btw, another advantage of a message-based interrupt is that the interrupt can span userid differences (while signals do not). This is more of an issue in a gateway environment since services like JupyterHub will launch the entire notebook server as the target user (retaining userid matches between kernel managers and kernel instances).

ccordoba12 · 2017-10-27T16:27:49Z

This is a really great addition!

I just have one question: Is it possible to add a restart message too? If that's the case, then we (Spyder) wouldn't need to wait for kernel nanny either.

filmor · 2017-10-30T11:39:52Z

@ccordoba12 What would you expect the kernel to do when it gets this message?

@takluyver There doesn't seem to be any activity on the ML regarding this PR, maybe it helps if you just cc all relevant people here ...

minrk · 2017-10-30T12:05:39Z

I think this is a fine addition. We just need to add this to the docs and bump the minor-revision of the spec.

ccordoba12 · 2017-10-30T21:44:41Z

@filmor, kernels can be restarted after a shutdown:

https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/manager.py#L293

This would be useful for Spyder because people could interact with external kernels the same way as they would do with kernels started by Spyder itself.

@minrk, what do you think?

SpencerPark · 2017-10-30T21:57:43Z

@ccordoba12 The shutdown_request looks like it already implements what you are asking about. The contents of this message has a restart flag to check if the shutdown precedes a restart.

What additional information are you looking for on top of that?

ccordoba12 · 2017-10-30T21:59:36Z

Sorry, it seems you're right. I'll take a look at it more carefully.

takluyver · 2017-10-31T13:50:10Z

Thanks @filmor . I think the remaining thing to do is to bump the protocol version to 5.3. The places I'm aware of this are in the messaging doc near the top, and in _version.py.

filmor · 2017-11-07T14:33:56Z

I have no idea why this test fails ...

- interrupt_mode="signal" is the default and current behaviour - With interrupt_mode="message", instead of a signal, a `interrupt_request` message on the control port will be sent

filmor · 2017-11-13T10:17:21Z

And I can not reproduce it locally. Any hints?

takluyver · 2017-11-13T13:13:15Z

It might be related to a change in zeromq, perhaps. @minrk this is the failure:

    def test_tracking(self):
        """test tracking messages"""
        a,b = self.create_bound_pair(zmq.PAIR, zmq.PAIR)
        s = self.session
        s.copy_threshold = 1
        stream = ZMQStream(a)
        msg = s.send(a, 'hello', track=False)
        self.assertTrue(msg['tracker'] is ss.DONE)
        msg = s.send(a, 'hello', track=True)
        self.assertTrue(isinstance(msg['tracker'], zmq.MessageTracker))
        M = zmq.Message(b'hi there', track=True)
        msg = s.send(a, 'hello', buffers=[M], track=True)
        t = msg['tracker']
        self.assertTrue(isinstance(t, zmq.MessageTracker))
>       self.assertRaises(zmq.NotDone, t.wait, .1)
E       AssertionError: NotDone not raised by wait

takluyver · 2017-11-13T13:50:18Z

I think Min's changes in #304, which was just merged, probably fixed this. Closing and reopening to test.

filmor · 2017-11-13T13:57:53Z

Yep, this looks better. Is there anything else to do?

takluyver · 2017-11-13T14:10:22Z

docs/kernels.rst

+- **interrupt_mode** (optional): May be either ``signal`` or ``message`` and
+  specifies how a client is supposed to interrupt cell execution on this kernel,
+  either by sending an interrupt ``signal`` via the operating system's
+  signalling facilities (e.g. `SIGTERM` on POSIX systems), or by sending an


SIGTERM -> SIGINT

takluyver · 2017-11-13T15:21:09Z

Thanks!

takluyver · 2017-12-15T12:48:35Z

@meeseeksdev backport

lumberbot-app · 2017-12-15T12:48:37Z

There seem to be a conflict, please backport manually

…the control port In Erlang I have only on POSIX and only in the newest version some control over the normal runtime signals, in particular SIGINT. However, I can easily pick up and process the respective message on the control socket to interrupt execution. Can this functionality be added? If yes, I'd continue and document the functionality and add tests. Maybe it would also be good to have the option of not sending an actual signal (via `os.kill`) at all. Signed-off-by: Thomas Kluyver <[email protected]>

takluyver · 2017-12-15T13:02:48Z

Backported manually.

SpencerPark mentioned this pull request Oct 13, 2017

Support interrupt request SpencerPark/IJava#2

Closed

filmor force-pushed the interrupt branch from 5a43bf0 to 05b2642 Compare October 19, 2017 09:16

SpencerPark reviewed Oct 19, 2017

View reviewed changes

filmor force-pushed the interrupt branch from 05b2642 to 16119c9 Compare October 19, 2017 18:14

filmor force-pushed the interrupt branch from 16119c9 to 42abb76 Compare October 30, 2017 15:46

filmor added 3 commits November 13, 2017 10:09

Configure interrupt mode via spec.

172d6cd

- interrupt_mode="signal" is the default and current behaviour - With interrupt_mode="message", instead of a signal, a `interrupt_request` message on the control port will be sent

Update docs.

f0e33ba

Bump protocol version.

21b9569

filmor force-pushed the interrupt branch from d748a48 to 21b9569 Compare November 13, 2017 09:10

takluyver closed this Nov 13, 2017

takluyver reopened this Nov 13, 2017

filmor changed the title ~~[RFC] Additional to the actual signal, send a message on the control port~~ Additional to the actual signal, send a message on the control port Nov 13, 2017

takluyver reviewed Nov 13, 2017

View reviewed changes

Fix signal name.

e2772bd

takluyver merged commit 0d7d00f into jupyter:master Nov 13, 2017

takluyver added this to the 5.2 milestone Dec 15, 2017

lumberbot-app bot added the Still Needs Manual Backport label Dec 15, 2017

takluyver removed the Still Needs Manual Backport label Dec 15, 2017

filmor deleted the interrupt branch April 6, 2018 13:03

filmor mentioned this pull request Aug 20, 2018

Interrupting the kernel does not work on Windows JuliaLang/IJulia.jl#503

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Additional to the actual signal, send a message on the control port #294

Additional to the actual signal, send a message on the control port #294

filmor commented Sep 22, 2017

SpencerPark commented Oct 7, 2017

rgbkrk commented Oct 8, 2017

takluyver commented Oct 9, 2017

SpencerPark commented Oct 9, 2017

takluyver commented Oct 10, 2017

filmor commented Oct 10, 2017

takluyver commented Oct 10, 2017

filmor commented Oct 18, 2017

filmor commented Oct 19, 2017

SpencerPark Oct 19, 2017

filmor Oct 19, 2017

takluyver commented Oct 20, 2017

lresende commented Oct 23, 2017

kevin-bates commented Oct 23, 2017

ccordoba12 commented Oct 27, 2017

filmor commented Oct 30, 2017

minrk commented Oct 30, 2017

ccordoba12 commented Oct 30, 2017

SpencerPark commented Oct 30, 2017

ccordoba12 commented Oct 30, 2017

takluyver commented Oct 31, 2017

filmor commented Nov 7, 2017

filmor commented Nov 13, 2017

takluyver commented Nov 13, 2017

takluyver commented Nov 13, 2017

filmor commented Nov 13, 2017

takluyver Nov 13, 2017

takluyver commented Nov 13, 2017

takluyver commented Dec 15, 2017

lumberbot-app bot commented Dec 15, 2017

takluyver commented Dec 15, 2017

Additional to the actual signal, send a message on the control port #294

Additional to the actual signal, send a message on the control port #294

Conversation

filmor commented Sep 22, 2017

SpencerPark commented Oct 7, 2017

rgbkrk commented Oct 8, 2017

takluyver commented Oct 9, 2017

SpencerPark commented Oct 9, 2017

takluyver commented Oct 10, 2017

filmor commented Oct 10, 2017

takluyver commented Oct 10, 2017

filmor commented Oct 18, 2017

filmor commented Oct 19, 2017

SpencerPark Oct 19, 2017

Choose a reason for hiding this comment

filmor Oct 19, 2017

Choose a reason for hiding this comment

takluyver commented Oct 20, 2017

lresende commented Oct 23, 2017

kevin-bates commented Oct 23, 2017

ccordoba12 commented Oct 27, 2017

filmor commented Oct 30, 2017

minrk commented Oct 30, 2017

ccordoba12 commented Oct 30, 2017

SpencerPark commented Oct 30, 2017

ccordoba12 commented Oct 30, 2017

takluyver commented Oct 31, 2017

filmor commented Nov 7, 2017

filmor commented Nov 13, 2017

takluyver commented Nov 13, 2017

takluyver commented Nov 13, 2017

filmor commented Nov 13, 2017

takluyver Nov 13, 2017

Choose a reason for hiding this comment

takluyver commented Nov 13, 2017

takluyver commented Dec 15, 2017

lumberbot-app bot commented Dec 15, 2017

takluyver commented Dec 15, 2017