git push timeouts results in excessive flux-write-sync tags #3075
Comments
coopernetes
added
blocked-needs-validation
|
When I first filed #2683, I think we talked about just making the tag have the branch name as the suffix rather than something pseudorandom... not sure if that would be a viable option or not. |
|
One other option I'm curious about is whether we can enable an option to assume repository write access if read access is available and if there are any exceptions to just defer them to when you are pushing the HelmReleases or whatever. We don't have to make this the default, but I have gotten some complaints about how often we are pushing this flux-write-check tag. |
|
I'm coming here with the same problem. My GIT is holding number of those tags. Is it safe to delete them? |
|
@Marx2 have you tried deleting them? I'm currently in the same situation, got 32k tags but I'm not sure if this would cause issues |
|
Yes, I've deleted them. After Flux upgrade and some configuration changes I don't remember now, it has not appear anymore |
|
That is an excessive number of tags and sounds like something important to fix, since it must be quite a pain to delete them all. The flux-write-sync tags should be safe to delete. If I add a long delay to my git server, I should be able to reproduce? If I understand what is happening correctly, the timeout happens before the write check succeeds, and the write check tag is not cleaned up since there is no indication that creating it was successful (but the git server eventually catches up, creating the tag, and this repeats creating those tags over and over until one write check is able to succeed within the timeout period.) |
|
We got less of those tags when increasing the timeout. Deleting the tags is safe. Run this in the repo: #!/usr/bin/env bash
while read tag; do
git push --delete origin $tag
git tag --delete $tag
done < <(git tag --list 'flux-write-check-*')
|
|
This can be reopened if there is a non-breaking change suggested that we can implement to make this kind of catastrophic overtagging less likely. A simple process was provided to help delete the extraneous tags without RSI, barring any contributor has a suggestion for how to make it better, I'm going to close this issue for now. Thanks for responding, thanks for using Flux! |
Describe the bug
Original thread on Slack: https://kubernetes.slack.com/archives/CBT6N1ASG/p1589473771368100
When flux is attempting to push the write-sync tag during sync-loop, if the upstream git server doesn't respond in a reasonable time, a new tag is created due to the pseudo-randomization introduced in #2684 and new attempts to push the sync tag result in an additional tag per sync interval. In large environments with multiple clusters and Flux deployments watching the same repo, this can result in an explosion of randomized git tags. In my situation, the upstream repo received 8000
flux-write-sync-<randombits>tags!To Reproduce
This is a bit tricky to reproduce since it was originally an issue with the upstream git repo (GitHub Enterprise). A fairly standard installation was used and the repo used for synchronizing Kubernetes manifests was Kustomize based with only a handful of resources (NetworkPolicy, about a dozen). Included flux deployment args below:
Expected behavior
Not really sure how to change this behaviour except to always bail out on error & delete the created tag.
Logs
Additional context
The text was updated successfully, but these errors were encountered: