allow node drain to block on batch allocs

[dvusboy]

It would be useful to support letting batch tasks finish when node-drain is enabled on a node. This can be handled with a new flag to nomad node-drain. When present it would:

• block new allocations to the node
• relocate service tasks to other available nodes
• terminate system tasks on the node

[olenm]

Do you mean to have 3 flags for node-drain which would allow the manual triggering of the bullet-points you listed?

[dvusboy]

Not really, I believe node-drain already does all 3 (I assume it does not "relocate" system tasks) actions and also relocate batch tasks. I'd like the flag to cause node-drain to only do these 3 actions, suppressing the relocation of batch tasks.

[stongo]

Is this being worked on? We are very motivated to add this feature but don't want to duplicate any efforts if this is already a work in progress. If not already in progress, we would most likely be willing to dedicate some engineering time to add this feature.

[dadgar]

@stongo No it is not currently being tackled. I would add a string DrainType that could be used to control the behavior of the drain. Then the next place to tackle would be generic_sched.go

[endocrimes]

Hey @dadgar,

I have a WIP patch that implements this feature for our deployment needs - But I'd like some opinions on how best to implement the UI for this in a way that works for the project itself.
How would you best want to expose this to users, only expose draining batch jobs, or also allow waiting for services?

[dadgar]

@dantoml Hey! So this is pretty good timing as @schmichael will be working on our next generation node draining strategies as part of Nomad 0.8! I am sure he would love to see your implementation so maybe linking the branch here would be a good start.

[endocrimes]

@dadgar Hey - The configurable version of our original hack patch is here: circleci@ddb347a - I'm happy to either fix this patch to be something mergeable, and/or contribute towards the 0.8 draining strategy work.

[endocrimes]

Relatedly - I'm not super sure what drain options make sense, outside of batch/(none|all) - I haven't used the services component of nomad, so I'm not sure of their mechanics.

[tgross]

Doing a little bit of issue cleanup and noticed this one has been long-done. From the [nomad node drain docs]:

Allocations for batch and sysbatch jobs will wait until they complete or the drain's deadline is reached, whichever comes first. These allocations will not be replaced.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.