You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.
Recently after careful increase of workloads in a cluster with the weave-flux-agent installed, we find that the weave agent (through the Weave Cloud Deploy dashboard) is timing out or takes longer time to load. During these period, we noticed that the number of go-routines on the flux agent go high and also cause the node on which Flux is running to run out of resources. Could we please help troubleshoot and understand the cause of this issue ?
To Reproduce
Install the latest version of the flux K8s agent (through Weave Cloud in this case).
Increase the number of workloads
Observe the flux agent golang Prometheus metrics on memory usage and goroutine count
Expected behaviour
I would not expect the memory of the fklux agent to grow beyond a reasonable level.
The text was updated successfully, but these errors were encountered:
@bricef Weave Cloud customers were upgraded to 1.13.2 today (new version range merged at 08:42 UTC).
The goroutines observed in this screenshot are probably due to the Kubernetes API rate-limiting requests, this will result in a pile of routines waiting for an answer and due to the deploy page making requests every x seconds, this pile will only grow. I implemented a timeout in #2171 (>=1.13.1) to prevent this from happening, so please check with the reporter if the issue is still relevant.
Describe the bug
Recently after careful increase of workloads in a cluster with the
weave-flux-agent
installed, we find that the weave agent (through the Weave Cloud Deploy dashboard) is timing out or takes longer time to load. During these period, we noticed that the number of go-routines on the flux agent go high and also cause the node on which Flux is running to run out of resources. Could we please help troubleshoot and understand the cause of this issue ?To Reproduce
Expected behaviour
I would not expect the memory of the fklux agent to grow beyond a reasonable level.
The text was updated successfully, but these errors were encountered: