feat: add commandRunnerCheck to healthcheck detail #6346
Conversation
| public HealthCheckResponseDetail check(final HealthCheckAgent healthCheckAgent) { | ||
| return new HealthCheckResponseDetail( | ||
| healthCheckAgent.commandRunner.checkCommandRunnerStatus() | ||
| == CommandRunner.CommandRunnerStatus.RUNNING); |
There was a problem hiding this comment.
Do we understand what causes the command runner to transition into ERROR state in practice? From looking at the code, it looks like this happens if the command runner has been unable to poll for 15 seconds (by default). Do we have a sense of how often this happens in practice, and why? It'd be good to have some assurance that this is rare, and that when it happens it singles an actual issue rather than something spurious, else failing the healthcheck because of it would not provide useful signal to users.
BTW, how was the 15 second default time limit chosen?
There was a problem hiding this comment.
The 15 seconds is kind of arbitrary. A normal poll of the command topic by default is 5 seconds so just 3x that was the rationale.
So there's two conditions that can lead to returning an ERROR, if there's currently a command being processed or not.
If there's no command being processed, we check to see if the command runner has been unable to poll for 15 seconds. If it hasn't, it means that the CommandRunner thread is most likely dead so the server would still be running without a CommandRunner thread, which is an unhealthy state (this is if the UncaughtExceptionHandler isn't enabled).
If there's a command being processed, and it's been processing for longer than ksql.server.command.blocked.threshold.error.ms that's also an unhealthy state for the server since it's probably stuck executing a command (this would most likely be a bug in the code and this wouldn't eventually finish executing on its own) and isn't making progress on the command topic
Description
#6308
Exposes command runner health in the healthcheck endpoint
Testing done
Issued a request to healthcheck endpoint
Unit test
Reviewer checklist