Nomad is re-computing templates every 5s following a variable change #24638
Comments
|
I've searched for other similar issues and found this one: #20618 — unsure if it's caused by the same root issue or if it's slightly different for similar symptoms. |
|
I've managed to reproduce the issue on my local system, using Nomad's dev agent installed through Homebrew at version Here are the reproduction steps (reproducing it takes 1 minute).
nano edge.hcljob "edge" {
group "edge" {
count = 1
task "edge" {
driver = "docker"
config {
image = "nginx:1.27-alpine"
}
template {
data = <<EOF
var={{ with nomadVar "nomad/jobs/edge" }}{{ .TEST }}{{ end }}
EOF
destination = "/local/first.txt"
change_mode = "signal"
change_signal = "SIGHUP"
}
template {
data = <<EOF
Static template
EOF
destination = "/local/second.txt"
change_mode = "signal"
change_signal = "SIGHUP"
}
}
}
}nano edge.nv.hclpath = "nomad/jobs/edge"
items {
TEST = "plop"
}
nomad agent -dev
nomad var put -in hcl @edge.nv.hcl
nomad job run edge.hcl
To be noted that if I remove the second template named |
|
Hi @valeriansaliou! This does indeed look like the same set of symtoms as #20618. I'm going to close this as a duplicate (just because it makes tracking easier) but cross-link this issue over there for @mismithhisler, who's actively working on that issue. |
Nomad version
Operating system and Environment details
Issue
I'm running a job with NGINX (called
edge). Multiple templates are rendered in this NGINX job, one of them contains NGINX upstreams pointing to other Nomad jobs (eg. a job calledweb), excerpt:Whenever I trigger a new allocation of the task
webwhich is depended upon NGINX, the service port will change, triggering a template re-render on the NGINX task. NGINX traffic is effectively steered to the newweballocation. So far so good.However, Nomad will immediately start checking for template diff every 5s on all template blocks of my
edgejob, inducing elevated CPU load on the Nomad client, with the CPU bursting every 5s. Nomad logs indicate that the templates are being checked for diff.To fix this infinite template diff occurring, I can either do:
edgejob (that contains the diffed templates), and re-submit it to Nomad from my Nomad CLI;Reproduction steps
Expected Result
The template will be re-rendered only once, or at least it'll converge to a state where no re-rendering will occur after a few seconds.
Actual Result
edgejob shows "Template re-rendered" once in the running allocation. It will only get reloaded once. This is all right & expected.edgejob generate, and it seems to re-schedule a template diff check task 5s later, again and again. The diff seems to yield to difference, which is proven by what's described in 1.Nomad Client logs
Every 5s, I am getting those logs on the Nomad client:
Job file
I made my job file for
edgeshorter than it is, by trimming some template contents:The text was updated successfully, but these errors were encountered: