Periodic dispatcher

[dadgar]

This PR adds support for running periodic jobs. It provides an at-most once run guarantee even across leadership transitions.

A follow up PR is needed to force run a periodic job.

[c4milo]

Awesome work @dadgar, I'm going to test this PR as soon as possible.

[ranjib]

When periodic jobs are created (i.e. nomad run periodic.hcl), the CLI does not show evaluation details similar to batch or service jobs. Is it expected? Similarly status for jobs also show , i was expecting that to show either next trigger time or something else, and probably previous allocations etc.

[dadgar]

@ranjib: It is expected that the CLI doesn't show eval details on periodic jobs. This is because periodic jobs never have an evaluation associated with them. They will spawn child jobs that will have evals but they themselves never do.

As for nomad status, that can be improved. This PR was just getting too big.

[ranjib]

@dadgar 🎉 One more thing, system jobs cant be periodic? like we cant use nomad to run chef periodically on all hosts?
FYI i took this branch for a spin under the [systemd driver] and its working well.

[dadgar]

@ranjib: Currently (and for the foreseeable future) there will not be periodic system jobs. You could probably create the behavior by registering a system job and on the interval you need to run chef, do an update to the job that would force Nomad to rerun the job (changing args would do it).

[c4milo]

I started testing this branch using the exec driver, but I've been noticing something weird with the allocations. For some reason once the task finishes, the allocation doesn't seem to be released. And, after a while, I start getting messages about resources being exhausted. I'm running locally using Vagrant and the VM is pretty much available for any workload.

[dadgar]

@c4milo: Yeah that isn't actually a problem with this PR. It is a bigger problem in general. We aren't freeing up the resources consumed by an alloc even when it is terminal in certain cases. This will be fixed by a PR I am working on now. Should be done soon.

[armon]

Given this behavior, maybe it should just be called "fixed" since it runs at fixed times. Otherwise, maybe pick something less likely to be accidentally used, e.g. "_internal_test"

[c4milo]

👯

[c4milo]

This is a beautiful christmas gift 🎄

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.