source-controller non-leader replica left in not-ready state #837
Comments
|
crickets... |
|
Hey @gladiatr72, have you noticed anything else in the logs in terms of errors? My first guess would be to check the resource limits (CPU and Memory) and resource saturation, as I have seen they causing the controller to misbehave without a proper reason - or reasonable error messages. Would you also be able to share what type of sources (and quantity) you have configured in your setup? |
|
Sure ⇶ k get deployments.apps -n flux-system source-controller -o jsonpath={.spec.template.spec.containers[0].resources} | jq -M .
{
"limits": {
"cpu": "1",
"memory": "1Gi"
},
"requests": {
"cpu": "50m",
"memory": "64Mi"
}
}The controller is not misbehaving. From the referenced ticket it is working as intended. The issue is in regards to the fluxv2 source controller being the only fluxv2 controller that uses a failed readiness check to manage leadership; whereas the kustomize, helm and notification controllers' secondary pods con ready, play a quick leadership game then slip into a passive state until the next election. I understand why those bits might not have been added to the source-controller yet, but #326 leaves one with the impression that it is not being considered as a thing that needs doing. |
|
The other part of what you asked for: current environment has 3 git sources and 1 helm repository with ~2 dozen charts |
|
If we would allow for standby pods to become ready, then consumers like kustomize and helm controller will not be able to fetch the source artifacts, standby pods don’t replicate the storage from the primary but kube proxy will randomly route calls to them. |
jonasbadstuebner
added a commit
to jonasbadstuebner/fluxcd-community-helm-charts
that referenced
this issue
Dec 14, 2023
fluxcd/source-controller#837 (comment) Signed-off-by: Jonas Badstübner <[email protected]>
jonasbadstuebner
added a commit
to jonasbadstuebner/fluxcd-community-helm-charts
that referenced
this issue
Dec 14, 2023
fluxcd/source-controller#837 (comment) Signed-off-by: Jonas Badstübner <[email protected]>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
in reference to: #326
Ok, but source-controller is the only flux component that actually does this. I thought that was the whole purpose of the leader election thing...
Personally, I don't care about the prom alerts. Those can be silenced. Filling up the event log with reams of
flux-system -- 0s -- Warning -- Unhealthy -- pod/source-controller-579b5cc8c9-2mvw8 -- Readiness probe failed: Get "http://10.64.183.82:9090/": dial tcp 10.64.183.82:9090: connect:; however, is not. (particularly on managed clusters where event retention is non-configurable (lifespan of a mayfly in a hurricane))helm-controller initial logs:
dxg6n
mcjk4
The text was updated successfully, but these errors were encountered: