Pre/Post hooks for tasks

[sheerun]

Hey,

I'd like to setup environment for a task before running it (for example mount an nfs volume). I'm using raw exec task driver. What is the best to do so with nomad?

Basically I need a way to run tasks in sequence instead in parallel.

Adam

[diptanu]

@sheerun We are thinking to introduce pre and post life cycle hooks in tasks so that would address your requirement. Currently, you will have to wrap your executable in a script and do the things you would like to do in that before the actual process starts.

[dadgar]

Converting the title to be about Pre/Post hooks

[sheerun]

An even better thing would be pluggable driver in form of script / binary that execs given command in specified environment.

[camerondavison]

Just to add a use case. I like to notify slack when my service starts and stops. I can do this by running a scripts, so I am not proposing integrating with slack (maybe if there was a way to just kick off a periodic task on start and stop of a task?)

[mortaliorchard]

Hi. Any updates on those two features? I could see both being very useful for security and monitoring systems.

[dudemcbacon]

Also looking for an update. This would be extremely useful for us.

[dadgar]

Hey folks, this is definitely on the roadmap! The ticket will be updated when the feature lands

[Kemicalish]

Will it be pre/post hook both for running/stopping task? (to clean some /tmp stuff for example) mainly with exec drivers

[dadgar]

Yeah exactly, pre-start, post-start, pre-stop, post-stop

[Kemicalish]

Also (maybe I am a bit greedy...) could a hook point to an other job? If it can be recursive it could enable a kind of simple and self managed dependency system.

[dadgar]

Haha that is getting greedy 😛 You could accomplish that with a pre-start that uses Nomads API or wait for first class DAG support which is also on the roadmap

[Kemicalish]

@dadgar Yeah i was thinking about that kind of workaround ^^ Thanks for the quick replies :)

[kurochan]

I am looking forward to this feature. When do you plan to release?

[dadgar]

@kurochan This work has not been started yet so no release date to give you!

[dreh23]

Is #2767 a viable route to go? If the maintainers can comment on it - I could help @pondohva to get this in shape. Although I believe this should have a different interface. Something more like this:

task {
   pre {
      command = "myprescript"
      args = ["-x", "-y", "-z"]
      artifact ...
   }

   config {
      command = "redis-server"
   }

   post {
      command = "mypostscript"
      args = ["-x", "-y", "-z"]
      artifact ...
   }
}

[agadelshin]

@dreh23 separate pre/post interface looks good.

[hectorj]

Would these commands be executed inside or outside of the task's container ?

I would love to be able to specify a full task config (driver etc.) for these hooks

[blalor]

rkt (or at least the now-deprecated ACI spec) does have support for lifecycle hooks along these lines. I think this concept makes a lot of sense for the raw_exec driver, but I'm having a hard time envisioning how it would work in the general sense in the context of rkt or Docker containers. Performing tasks on the host when using the rkt driver would make for some trippy isolation cases.

[agadelshin]

I think we need to run commands inside container, there are cases where we need to run DB migrations (e.g. django applications), or deregister from cluster/monitoring via post-hook

[dvusboy]

Within a TaskGroup, there is no explicit ordering (other than the new leader Task), it makes 'sub-task' like setting up (or tearing down) a Docker volume or volume container, network, etc. difficult to do with just a Nomad job specification. My current work around is to use raw_exec and docker-compose. A native solution in Nomad would be great.
I'm not sure pre and post hooks would be sufficient. What about handling error conditions? This is tricky business and this discussion is great.

[dreh23]

To clarify our use case:
We need to execute certain commands before and after a container is spawned. We are dynamically mounting USB Hardware onto the node and need to register before and deregister after the container came up/down. This has to be on the node itself and not inside the container. Now we are wrapping everything in a script, but this is error prone and an even greater hack.

[stevenscg]

I could see deployment hooks being very useful as well. In our old ansible-based deployment model, we had (ansible) tasks that reported the deployment to our chat and metrics systems.

[nanoz]

ContainerPilot has an interesting take on task ordering, based on lifecycle events.

https://github.com/joyent/containerpilot/blob/master/docs/10-lifecycle.md#how-can-jobs-be-ordered

[pims]

Any update on this feature request? Or the custom driver feature?

[yellowmegaman]

Also would be happy to see this live.

Got very simple and important use case - flush cassandra database before shutdown.

[eggsbenjamin]

bump

[rkettelerij]

Another use case where this would be useful:

• Say you're using the template stanza to get a certificate from Vault and render that certificate to a PEM bundle.
• Now you want to pass that PEM bundle to a JVM-based system like Kafka of Elasticsearch in order to enable TLS. The JVM can't handle raw PEM files, it expects a Java keystore (JKS). So you need to do a PEM->PKCS#12->JKS conversion.
• A pre-hook would be useful here. Everytime the certificate in Vault expires, the template stanza is triggered, it renders a new certificate, and signals the task. Kicking off the certificate conversion in the pre-hook and eventually triggering the actual task. In this case: starting Kafka with the given JKS files.

[nvx]

I stumbled upon this while looking for an option to manage Vault PKI issued certificates into a Java Keystore.

For my usecase though a pre-start hook isn't good enough as this wouldn't handle the credential changing. For this usecase a "post-template" hook would be ideal (executes after the template file is rendered to disk, but before the task is started or change_mode action is performed).

[bkannadassan]

compose

I am using the same but now how does nomad monitor each docker started with compose ?. How are you managing restarting the same ?.

[yishan-lin]

I think it's worth relating this issue with #419, as the implementation we are considering at the moment is indeed via pre/post hooks to enable running tasks in a predefined sequential order.

[eveld]

Will it be possible to mix types? Like have a batch task and a service task together? It always feels a bit nasty running a service task to run a 1-off script and artificially keeping it around with tail -f /dev/null so the whole job doesnt get killed. Greetings, Erik

…

On Fri, Oct 25, 2019, 01:55 Yishan Lin ***@***.***> wrote: I think it's worth relating this issue with #419 <#419>, as the implementation we are considering at the moment is indeed via pre/post hooks to enable running tasks in a predefined sequential order. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1061?email_source=notifications&email_token=AAFLNGKEHCMNXMAC54KVSMDQQIYXJA5CNFSM4CANLMBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGYOOI#issuecomment-546146105>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFLNGJJ67KK4TYRUNCQP3DQQIYXJANCNFSM4CANLMBA> .

[sagarrakshe]

I faced the similar issue in our deployments, so I created a tool.
https://github.com/sagarrakshe/nomad-dtree

[tgross]

Shipped in 0.11.0-beta1

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.