Add retry_for_client_errors option to outputs.http plugin

[mzbroch]

Feature Request

Proposal:

Add retry_for_client_errors option to outputs.http plugin

Current behavior:

Telegraf's outputs.http plugin retries writing the metric in case of 4xx errors. No configurable parameters exist to control this current behaviour.

Desired behavior:

outputs.http plugin in Telegraf understands client side error codes and can optionally not retry (drop the metrics) in case of 4xx errors.

Use case:

• Allows users to drop the metric in case of client side errors while writing the metric with HTTP plugin.
• Solves Prometheus remote write with Thanos out of order metrics stops metrics processing #9365

[dgarros]

Our team would be happy to send a PR to add this flag but it would be helpful to understand if this would be accepted first.

[rabenhorst]

WDYT about adding a client_errors_ignore_list option that takes an array of 4xx status codes so you can flexible configure which error codes should be ignored in the output (i.e. return no error).

[mzbroch]

It sounds reasonable, does this mean we're making a project decision about this issue?

[nvinzens]

Opened a PR for that in #10186, happy to change the naming if peeps don't agree with non_retryable_statuscodes.

[DStrand1]

Closed by #10186