Improvement: add metrics to prom client

[OGKevin]

This pr add pkg/extprom/http/instrument_client.go with code to instrument an HTTP client.
The approach is to add metrics clients to the HTTP config.
Not sure if this is the right approach or if another clever design can be used, this was the most obvious and backward compatible way I could come up with.

fixes: #4545

Signed-off-by: Kevin Hellemun [email protected]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Verification

# HELP thanos_rule_query_http_client_in_flight_requests A gauge of in-flight requests.
# TYPE thanos_rule_query_http_client_in_flight_requests gauge
thanos_rule_query_http_client_in_flight_requests 0
# HELP thanos_rule_query_http_client_request_duration_seconds A histogram of request latencies.
# TYPE thanos_rule_query_http_client_request_duration_seconds histogram
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.005"} 0
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.01"} 1
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.025"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.05"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.1"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.25"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="0.5"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="1"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="2.5"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="5"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="10"} 2
thanos_rule_query_http_client_request_duration_seconds_bucket{le="+Inf"} 2
thanos_rule_query_http_client_request_duration_seconds_sum 0.018484452
thanos_rule_query_http_client_request_duration_seconds_count 2
# HELP thanos_rule_query_http_client_request_total 
# TYPE thanos_rule_query_http_client_request_total counter
thanos_rule_query_http_client_request_total{code="200",method="post"} 2

[bwplotka]

Nice! I love it. LGTM, just one minor nit.

[OGKevin]

The metrics now look like this:

# HELP http_client_in_flight_requests A gauge of in-flight requests.
# TYPE http_client_in_flight_requests gauge
http_client_in_flight_requests{client="alertmanager"} 0
http_client_in_flight_requests{client="query"} 0
# HELP http_client_request_duration_seconds A histogram of request latencies.
# TYPE http_client_request_duration_seconds histogram
http_client_request_duration_seconds_bucket{client="query",le="0.025"} 24
http_client_request_duration_seconds_bucket{client="query",le="0.05"} 24
http_client_request_duration_seconds_bucket{client="query",le="0.1"} 24
http_client_request_duration_seconds_bucket{client="query",le="0.5"} 24
http_client_request_duration_seconds_bucket{client="query",le="1"} 24
http_client_request_duration_seconds_bucket{client="query",le="5"} 24
http_client_request_duration_seconds_bucket{client="query",le="10"} 24
http_client_request_duration_seconds_bucket{client="query",le="+Inf"} 24
http_client_request_duration_seconds_sum{client="query"} 0.105682592
http_client_request_duration_seconds_count{client="query"} 24
# HELP http_client_request_total 
# TYPE http_client_request_total counter
http_client_request_total{client="query",code="200",method="post"} 24

[bwplotka]

Sweet, all good except two nits. Good job!

[bwplotka]

e2e fail looks unrelated, but let's double check. Retrying.

[bwplotka]

Retried, will monitor if this time e2e fails again or not.