Execute the MongoDB Connection Health Check on Startup

[Inithron]

Description

It would be great if the MongoDB Connection Health Check could be improved.

Current behavior

After startup following health check is displayed:

{
    "status": "UP",
    "checks": [
        {
            "name": "MongoDB connection health check",
            "status": "UP"
        }
    ]
}

But this status is misleading / wrong. Even if no database is available, the status is UP. Only when the application tries to store the first message in the DB, the status goes to down:

{
    "status": "DOWN",
    "checks": [
        {
            "name": "MongoDB connection health check",
            "status": "DOWN",
            "data": {
                "<default>": "KO, reason: Timed out while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: no further information}}]",
                "<default-reactive>": "KO, reason: null"
            }
        }
    ]
}

Improvement

It would be great, if the connection to the DB would be checked on startup and not when the application tries to store the first message in the DB.

Benefit

With this changed approach the pod in Kubernetes would never get green if there is a wrong connection string or the DB is down.

Implementation ideas

No response

[quarkus-bot]

/cc @geoand (kubernetes), @iocanel (kubernetes), @jmartisk (health), @loicmathieu (mongodb), @xstefank (health)

[geoand]

Makes sense IMO

[geoand]

Looking at the code, it should work as expected.

What version of Quarkus are you using?

[Inithron]

I am using 3.17.6
Here is a small reproducer with a unit tests showing the mentioned behavior:

reproducer-for-45482.zip

[geoand]

Thanks!

[geoand]

#45523 fixes the issue

[Inithron]

Hi @geoand,
I tested the fix with 3.17.8 but it still seems not to be correct. When I execute q/health it shows now the correct status ("DOWN" if no DB is available). But now each call of q/health takes 30 seconds (which is the default value for mongodb.server-selection-timeout ). But the default timeout in Kubernetes for the readiness and liveness check is 1 second, see timeoutSeconds. So from my point of view q/health should not block the caller. Maybe a background task is necessary to check the health of the DB regularly wich caches the status. And if q/health is called, the cached value can be returned. What do you think about this?

[geoand]

I would need a sample application that behaves as you describe to be able to figure out what's going on

[Inithron]

I updated the reproducer to version 3.17.8 and disabled #quarkus.mongodb.server-selection-timeout=1 so that the default timeout is used:
reproducer-for-45482_2.zip

1. After starting the application you can see in the logs nothing is happening (no logs about MongoDB).
2. Wenn you now open http://localhost:8080/q/health/ready it took round about 30 seconds until this page is loaded.
3. After the call http://localhost:8080/q/health/ready you can see the first time logs from MongoDB (INFO [org.mon.dri.client] (vert.x-worker-thread-1) MongoClient with metadata [...] and the error: ERROR [io.qua.mut.run.MutinyInfrastructure] (executor-thread-1) Mutiny had to drop the following exception: com.mongodb.MongoTimeoutException [...])
4. Every refresh in the browser or new call to http://localhost:8080/q/health/ready took again 30 seconds.
5. http://localhost:8080/q/health/live is loaded without noticeable delay. I expect the same for http://localhost:8080/q/health/ready even if no DB is available.

[geoand]

@xstefank please take a look at ^

[xstefank]

Hi @Inithron, I understand what you are after but the problem described in this issue is really fixed by @geoand's PR. Basically, the Mongo health check is designed in this way to wait for the timeout to run out. But personally I don't see anything wrong with your idea, I just moved it to a new issue, since it's really a new feature - #45924. If no one will object, I can implement it.