[Heartbeat] Adjust State loader to only retry for failed requests and not for 4xx

[vigneshshanmugam]

Summary

Heartbeat uses state loader to get the last status from the ES cluster and loads the current monitor state after the monitor has been successfully ran. The ES loader has a backoff of 3 retries which will be attempted when the request fails due to connection failures or ES being unavailable. But currently, we are also retrying for 4xx where the API key has limited permissions to read the ES state.

This consequently affects the time a particular execution is taking when we are running inside the SAAS service.

Proposal

Retry ES state loader requests only for network connection failures or 5xx and avoid retrying for 403 and 4xx when we knows its a valid status from the ES side.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[vigneshshanmugam]

Notes:

• ES wont return 404 for the state requests since we are using _search query.
• We need a mechanism to parse the response bytes for these unauthorized requests and short circuit the retried requests.
• We also need to do perform this _search query only once for failed requests, We should not be updating the monitor state when we already know the user is not authorized.

[martinscholz83]

@vigneshshanmugam, I could help if you want? Currently the error is a json string which could be unmarshalled and then check the status. For example

"401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"

But what about errors e.g. when the server is offline or any certificate issues. Then the error message has a different structure.

"couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://localhost:9200: Get \"https://localhost:9200\": dial tcp 127.0.0.1:9200: connect: connection refused]"

[martinscholz83]

I guess you had this attempts in mind?
https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L355

[devcorpio]

Hi @martinscholz83,

We started progressing in this issue a while ago. We overlooked your comments. Sorry about this.

We strongly appreciate your interest in solving this!

I guess you had this attempts in mind?

The attempts in mind were related to the query that extracts the state.

The one you mentioned was related to the "initial" connection to ES.

Thanks again,
Alberto