Separate liveness and readiness checks

[ukutaht]

Changes

Adds two new endpoints:

1. /api/system/health/live does nothing more than return 200 OK with a JSON body {"ok": "true"}
2. /api/system/health/ready checks all the necessary dependencies for accepting traffic (identical to old /api/health behaviour):
   2.1 Postgres reachable
   2.2 Clickhouse reachable
   2.3 Ingestion caches ready

The old /api/health endpoint is still usable to give time to migrate external checks.

Technically ingestion can keep working as long as caches are loaded but postgres is unreachable. Future possibility is separating ingestion readiness check from app readiness.

[aerosol]

What decision(s) are we willing to make based on HTTP availability alone? IMHO it's the cache(s) readiness check that should be separated.

BTW, omission on my part, only two caches are currently checked against. We should include user agents, sessions, country rules, page rules, hostname rules too.

Checking cache size is not mandatory, but convenient way of making sure there's no exit thrown out of a key lookup, at least. Without those marked green we should not expose any node via LB.

[ukutaht]

What decision(s) are we willing to make based on HTTP availability alone?

The readiness probe is used to decide whether to route traffic to the container or not.
The liveness probe is used by kubernetes to decide whether to restart a container or not.

As I understand it the liveness probe should only fail in fatal scenarios that the application cannot recover by itself. Database downtime or BEAM process crashes, for example, should in theory be recovered without a restart which is the main idea with this separation.

BTW, omission on my part, only two caches are currently checked against. We should include user agents, sessions, country rules, page rules, hostname rules too.
Checking cache size is not mandatory, but convenient way of making sure there's no exit thrown out of a key lookup, at least. Without those marked green we should not expose any node via LB.

I'll add all the caches to the readiness probe.

[ukutaht]

@aerosol from your basecamp comment

Happy to work on any /api/health improvements allowing us to achieve better orchestration. Basically my take is we should start by refining our restart strategy to „let it crash more before we take it down”, to have more meaningful traces of any underlying conditions.

This is exactly what the separate liveness probe does. The app will only be restarted when the container is unreachable or is reachable and cannot even respond with 200 OK.

[aerosol]

@ukutaht yes, but given we let the traffic in once caches are initially ready, which /correct me if I'm wrong/ is currently not enforced? cc @cnkk

[ukutaht]

Sorry I don't understand that sentence @aerosol

[aerosol]

Sorry I don't understand that sentence

Sorry, hope all cleared up in basecamp. I think we might give it a go. Please consider merging #5057 - not critical, mostly cosmetic though.