Flux syncs git repo on a 5 minute timer, regardless of --git-poll-interval setting

[pmarkus]

We're running a self-host Flux, version 1.2.5.
When setting the Flux daemon's --git-poll-interval to 5s. Our expected result would be that Flux tries to update the state of Kubernetes within 5 seconds of a commit being pushed to the git repo.

It seems however, that Flux only applies changes from the remote git repo to Kubernetes on a set 5 minute interval even if we set the --git-poll-interval lower. So if we push something to the git repo, it can take anywhere up to 5 minutes before that change is applied in Kubernetes.

--git-poll-interval will still controll how often Flux applies updates to Kubernetes from it's internal copy of the git repo. That is, we can see that if we set --git-poll-interval to 25 seconds and manually kubectl delete a deployment controlled by Flux, Flux will make sure to reapply that deployment within 25 seconds.

[squaremo]

Thanks for the bug report! Fetching new commits from the upstream repo was rewritten in #940, and at that point the git polling interval was hard-wired (but not the syncing interval, which still follows the --git-poll-interval flag).

There's no principled reason for this; it was left out of the rewrite in anticipation of being able to use webhooks instead. However, the hooks aren't there yet, so we could at least reinstate the ability to adjust the git polling interval.

[petervandenabeele]

Is this issue actually resolved in 1.5.0 of Flux ?

I just added this to the flux-deployment and re-applied it.

         # replace or remove the following URL
-        - [email protected]:petervandenabeele/flux-example
+        - --git-url=git@gitlab_______:kubernetes/flux-deploy-edge
         - --git-branch=master
+        - --git-path=edge
+        - --git-poll-interval=120s

It restarted the replicaset and the pod 6 minutes ago, but no new polling ...

➜  flux git:(master) ✗ kubectl logs  $(kubectl get pods -n flux | grep flux | cut -d' ' -f1) -n flux | grep -v 'HTTP 429' | grep -v 'nvidia' 
ts=2018-08-23T14:26:21.98916395Z caller=main.go:140 version=1.5.0
ts=2018-08-23T14:26:22.009735921Z caller=main.go:229 component=cluster identity=/etc/fluxd/ssh/identity
ts=2018-08-23T14:26:22.009773332Z caller=main.go:230 component=cluster identity.pub="ssh-rsa AAAAB3N..." =>  I pushed this key around 14:29:00
ts=2018-08-23T14:26:22.009796073Z caller=main.go:231 component=cluster host=https://10.39.240.1:443 version=kubernetes-v1.10.5-gke.4
ts=2018-08-23T14:26:22.009848483Z caller=main.go:243 component=cluster kubectl=/usr/local/bin/kubectl
ts=2018-08-23T14:26:22.012107412Z caller=main.go:251 component=cluster ping=true
ts=2018-08-23T14:26:22.016453003Z caller=main.go:370 url=git@gitlab__________:kubernetes/flux-deploy-edge user="Weave Flux" [email protected] sync-tag=flux-sync notes-ref=flux set-author=false
ts=2018-08-23T14:26:22.01655451Z caller=main.go:425 upstream="no upstream URL given"
ts=2018-08-23T14:26:22.018148449Z caller=main.go:442 addr=:3030
ts=2018-08-23T14:26:22.019067506Z caller=loop.go:87 component=sync-loop err="git repo not ready: git repo has not been cloned yet"  => THE KEY WAS NOT THERE
ts=2018-08-23T14:26:22.022161843Z caller=images.go:16 component=sync-loop msg="polling images"
ts=2018-08-23T14:26:22.022251071Z caller=images.go:22 component=sync-loop error="getting unlocked automated services: git repo not ready: git repo has not been cloned yet"
ts=2018-08-23T14:26:22.457536088Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.5.0
ts=2018-08-23T14:31:22.022290732Z caller=loop.go:87 component=sync-loop err="git repo not ready: failed to push some refs to '[email protected]:kubernetes/flux-deploy-edge'"  # IT CANNOT PUSH since kubernetes-deploy is `developer` role and that is not allowed to update tags (an initial creation of the `flux-write-check` tag had succeeded an hour ago).
ts=2018-08-23T14:31:22.022389826Z caller=images.go:16 component=sync-loop msg="polling images"
ts=2018-08-23T14:31:22.022420647Z caller=images.go:22 component=sync-loop error="getting unlocked automated services: git repo not ready: failed to push some refs to '[email protected]:kubernetes/flux-deploy-edge'"

[squaremo]

@petervandenabeele It won't start polling until it considers the git repo to be ready, and it won't consider it to be ready until it has demonstrated that it can write a tag to the upstream repo.

ts=2018-08-23T14:31:22.022290732Z caller=loop.go:87 component=sync-loop err="git repo not ready: failed to push some refs to '[email protected]:kubernetes/flux-deploy-edge'"

At present it needs to be be to write a tag, to complete a sync. This may be fixed in future development, possibly by using some other means to store the sync high-water mark.

(Reading slightly more closely, I see that a tag can be created with the given permissions; probably what is failing is deleting it again, which fluxd does to tidy up)

[petervandenabeele]

Thanks:

1. confirming your suspicion in your last sentence.

• Initially, with only Developer permissions, it could write that
  kubernetes-deploy pushed new tag flux-write-check to kubernetes/flux-deploy-edge
  (hours ago, at 15:25 CEST) but not update it afterwards
• now with Maintainer permissions, it can do
  kubernetes-deploy pushed new tag flux-sync to kubernetes/flux-deploy-edge

2. Confirming your initial statement: as soon as tag pushing problem is resolved, goes to 120 seconds cycle:

ts=2018-08-23T14:36:22.0225484Z caller=loop.go:87 component=sync-loop err="git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:36:22.022679232Z caller=images.go:22 component=sync-loop error="getting unlocked automated services: git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:41:22.022770465Z caller=loop.go:87 component=sync-loop err="git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:41:22.022931626Z caller=images.go:22 component=sync-loop error="getting unlocked automated services: git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:46:22.023051052Z caller=images.go:22 component=sync-loop error="getting unlocked automated services: git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:46:22.023105882Z caller=loop.go:87 component=sync-loop err="git repo not ready: failed to push some refs to 'git@gitlab.______:kubernetes/flux-deploy-edge'"
ts=2018-08-23T14:46:57.043696927Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=f59efbc41a0e49cbdcd5dac054fdaacedabb88b3
ts=2018-08-23T14:48:57.746333134Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=f59efbc41a0e49cbdcd5dac054fdaacedabb88b3
ts=2018-08-23T14:50:58.408675155Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=f59efbc41a0e49cbdcd5dac054fdaacedabb88b3
ts=2018-08-23T14:52:59.045986639Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=f59efbc41a0e49cbdcd5dac054fdaacedabb88b3
ts=2018-08-23T14:54:59.753695098Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=177896b49a397fd9a01d9440fa0afc8b3b386928
ts=2018-08-23T14:55:01.865560526Z caller=loop.go:100 component=sync-loop event=refreshed url=git@gitlab.______:kubernetes/flux-deploy-edge branch=master HEAD=177896b49a397fd9a01d9440fa0afc8b3b386928