queue: Ability to send different signals to running tasks

[aschuh-hf]

dvc queue kill can be used to terminate running tasks. One can also use dvc queue logs --follow to attach to the output of a running task, but CTRL+C (SIGINT) will only detach but not forward the interrupt signal to the running task (which is good). There is no option at the moment to gracefully interrupt a running task, e.g., a training stage using DVC checkpoints as one can and is being used in the user guide for DVC Experiments with checkpoints. Options for dvc queue kill which enable the user to send different signals (SIGTERM, SIGINT) to the running task similar to the Unix kill command would be useful.

Related, sending SIGINT via CTRL+C would be possible if there was a command to attach to a running task such that the signal is indeed forwarded to the task rather than handled by, e.g., dvc queue logs --follow. Possibly similar to docker attach, where one can declare --detach-keys to detach without interrupting the task. A key combination different from CTRL+C could also have been used for dvc queue logs --follow to unfollow / detach again without interruption (see also tmux) but CTRL+C would have been forwarded to the task still. This different behavior could be covered by a new dvc queue attach command (which could make dvc queue logs --follow obsolete actually).

[efiop]

@karajan1001 @pmrowla Would something like dvc queue kill --signal SIGINT make sense? WDYT?

[pmrowla]

Yes, sending sigint would work

[dberenbaum]

Why is sigkill the default? Should we default to sigint?

[karajan1001]

Why is sigkill the default? Should we default to sigint?

Sometimes, Ctrl-C does not stop the process while kill -9 always does it. Maybe we should add a --force flag to queue kill and make the default queue kill sending SIGINT and queue kill --force to send SIGKILL

[karajan1001]

@pmrowla , @dberenbaum
A problem is that the training process is running in a sub-process of the DVC command. Sending SIGINT to the DVC command couldn't stop the sub-process of the training (But SIGKILL can). This ticket is more complex than I used to believe. If we want to implement this we need to first record the sub-process id in some way and send the signal to that process.

[pmrowla]

Discussed this with @karajan1001, looks like there may be an issue with our SIGINT handling in dvc exp exec-run (which is used internally to run the exps from celery), so that's where we'll need to start for this issue. It should not be necessary for us to keep track of each child pipeline stage process ID.

[dberenbaum]

How important is this? It seems like a new feature request that may not be urgent.

Issues like #8612 should work even with SIGKILL. Can we hold off on this until we address the reported regressions and bugs?