Add sync status label to deployments #8622
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
deepthiskumar
added
the
ci-build-me
lk86
reviewed
Apr 14, 2021
automation/bake/Dockerfile
Outdated
| @@ -10,6 +10,8 @@ RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyri | |||
|
|
|||
| RUN apt-get update && apt-get install -y google-cloud-sdk | |||
|
|
|||
| RUN apt-get update && apt-get install -y kubectl | |||
There was a problem hiding this comment.
This should move to dockerfiles/Dockerfile-coda-daemon if it's going to be required for health checks
lk86
approved these changes
Apr 14, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new label
syncStatusto deployments whose value gets updated as part of the readiness health check. This label can then be used in grafana dashboard and/or in alerting rules to filter metrics from nodes with a specified sync status.Here's an example dashboard (in edit mode so you can see the queries) from a private testnet deployed with the changes from this PR.
https://o1testnet.grafana.net/d/qx4y6dfWz/network-overview?editPanel=27&orgId=1&refresh=1m&var-testnet=test-labels&from=now-30m&to=now
The initial status when deploying is
INITwhich then gets updated to one of following statuses of the daemon:CONNECTINGLISTENINGOFFLINEBOOTSTRAPSYNCEDCATCHUPShould this be off of develop?
I'm not sure if it is incompatible against existing deployments. In any case we'll want to gradually upgrade all the nodes of existing networks(mainnet/devnet)
After approved, I'll update the alert expressions as described in #8522