Bug 2013243: jssonet: ignore Alarm method in etcdGRPCRequestsSlow alerts #691
Conversation
openshift-ci
bot
added
the
bugzilla/severity-medium
|
@hasbro17: This pull request references Bugzilla bug 2013243, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
added
the
bugzilla/invalid-bug
|
/bugzilla refresh |
openshift-ci
bot
added
bugzilla/valid-bug
|
@hasbro17: This pull request references Bugzilla bug 2013243, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-4.9 |
|
@hasbro17: once the present PR merges, I will cherry-pick it on top of release-4.9 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
| description: 'etcd cluster "{{ $labels.job }}": Observed surge in etcd writes | ||
| leading to 50% increase in database size over the past four hours on etcd | ||
| instance {{ $labels.instance }}, please check as it might be disruptive.' | ||
| description: 'etcd cluster "{{ $labels.job }}": Observed surge in etcd writes leading to 50% increase in database size over the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.' |
There was a problem hiding this comment.
the way forward since v1.0.0 however seems to be non-wrapped lines so we should probably stick to that.
Yes, I think this formatting changed was due by @hexfusion maybe having an outdated gojsontoyaml install during recent changes? Maybe we can add another check for this here https://github.com/openshift/cluster-etcd-operator/blob/master/hack/generate.sh#L5?
There was a problem hiding this comment.
yeah I will keep that in mind but we should bake it into the script.
openshift-ci
bot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hasbro17, lilic The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/retest |
|
I have been thinking about this one a bit. While Alarm is not really an expected result of a health check the fact that it is slow should be the same as any RPC thus part of the alert? With defrag it made sense to omit because the duration of the request was not inline with the thresholds of the alert. But do we care so much which RPC is slow vs in general slow given there is no explicit reason for the slow request? In fact this helps us to identify Alarm as strange in the steady state. |
|
Closing this for now as we're not fully on board with specifically excluding Alarm from slow grpc requests given that it's still an indication that RPC calls in general are slow. |
|
@hasbro17: This pull request references Bugzilla bug 2013243. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
The cluster-etcd-operator readiness probe uses the
etcdctl endpoint healthcmd to check the health of the clustercluster-etcd-operator/bindata/etcd/pod.yaml
Lines 210 to 211 in ea4c530
As part of the endpoint health command, etcdctl checks the active alarms with grpc calls to the
Alarmmethod:https://github.com/etcd-io/etcd/blob/f2bc5eee916de63ac525ce5a843e849f73921415/etcdctl/ctlv3/command/ep_command.go#L139
To eliminate noise from the
etcdGRPCRequestsSlowalerts we should discount the effect ofAlarmgrpc calls which could end up being slow due to the repeated calls, and indicating this to the user is not useful.This is only done downstream through our
custom.libsonnetrules file for now.Side note:
Depending on when you installed
gojsontoyamlit may or may not wrap long lines in its output.The way forward since
v1.0.0however seems to be non-wrapped lines so we should probably stick to that. Note you may want to rungo get go get github.com/brancz/gojsontoyamlagain ifmake generateresults in wrapped lines for you.See:
/cc @lilic @hexfusion