change the status code which healthcheck API returns

[ShintaroOba]

Description
Healthcheck endpoint returns 503 (Service Unavailable) when server is unhealthy.

Use case / motivation
Current healthcheck api returns status code "200 (OK)"even if server is "unhealthy".
It couldn't be working healthcheck correctly when some cloud services (like aws) which can periodically sends healthcheck requests to airflow.
Because they are judging success or failure only seeing status code of api response. They don't see response body json at all.

Are you willing to submit a PR?
Yup. But I'm not ready ready PR for this issue.

Related Issues

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jhtimmins]

Confirmed that this is the behavior, as any failure is caught and then a standard response is returned.
https://github.com/apache/airflow/blob/master/airflow/api_connexion/endpoints/health_endpoint.py#L37

@ephraimbuddy @mik-laj @kaxil do y'all think this should be updated?

[ephraimbuddy]

Confirmed that this is the behavior, as any failure is caught and then a standard response is returned.
https://github.com/apache/airflow/blob/master/airflow/api_connexion/endpoints/health_endpoint.py#L37

@ephraimbuddy @mik-laj do y'all think this should be updated?

Yep @jhtimmins, it's a good call. Needs to be updated

[mik-laj]

this health check is part of the webserver and if the webserver returns an incorrect value, the error code will be different than 200.

For other components it doesn't work and we probably shouldn't fix it because it won't allow checkk component in isolation i.e. if the webserver won't work then all other components will also report that they are in unhealthy state, which won't be correct. I think, we should use other health check e.g. CLI command(preferred) or separate health check server per component (risky as this means that these component must accept incoming traffic)

[mik-laj]

Duplicate: #11161