Creating a Way to Pause Workers from Grabbing New Jobs

[boxbeatsy]

I'm currently using django-rq on Heroku. On Heroku, workers don't finish their current tasks when a shut down signal is received. Instead, they run for 10 seconds, and then terminate mid process.

We're currently working around this by waiting until all workers are idle before deploying new code (deploying new code causes workers to reboot).

However, one thing that I think would be immensely helpful would be to have a way to pause workers from grabbing new jobs. Even though we currently wait for queues to clear out before deploying, if a long-running task happens to be enqeued WHILE deploying, it gets killed.

[mbodock]

Hello @boxbeatsy, the behavior you describe in Heroku is similar with the Docker Engine.

First the container gets a SIGSTOP, if the container don't terminate in 10 sec, a SIGTERM is sent to the process.

The rqworker works a little different, your first gets a SIGSTOP/SIGQUIT(ctrl+c) that triggers a worm shutdown, which will wait the task finishes. If a new SIGSTOP/SIGQUIT is sent a cold shutdown begins, killing the process and failing the task.

I do not have experience with Heroku, but I think it fits in the first case.

And about a way t pause the workers, it's not entire necessarily once you can use the worm shutdown. Maybe there is a way to send Linux signals to your processes on Heroku and fix the behavior.

[boxbeatsy]

Hi @mbodock, thanks for the input. I believe that Heroku handles the SIGTERM signal differently which doesn't give the worker the ability to complete the task at hand: rq/rq#584 (comment)

That is, I don't believe the worm shutdown is honored on Heroku. Please let me know if I'm missing something though!

[selwin]

RQ actually already has an "rq suspend" command which is not documented. This command pauses job execution so you can pause workers before shutting it down in Heroku. We should make a management command to wrap this command in Django-RQ.

PR welcome 😃