Standby servers should not forward /sys/metrics calls

[tstoermer]

Describe the bug
Standby servers in a vault cluster are forwarding the "/sys/metrics" call to the active server.

To Reproduce
Steps to reproduce the behavior:

1. Setup vault in a cluster with one active and standby servers.
2. Enable prometheus metrics endpoint.
3. Scrape in Prometheus all vault servers.
4. Standby servers are forwarding the "/sys/metrics" calls to active server.
5. Metrics in Prometheus for standby servers are showing the same values as the active server.

Expected behavior
Standby servers should provide their own metrics instead of forwarding the metrics call to active server. This would make more sense from monitoring perspective.

Environment:

• Vault Server Version: 1.1.2 (Docker)

Vault server configuration:

• Vault Cluster with 1 active and 1 standby server
• Consul Cluster as Backend
• Enabled Prometheus endpoint

Additional context
Vault code looks like standby servers will always forward metrics requests to active server. There was also related discussion in vault group (I wasn't sure, if it was caused by a configuration error).

[mtse-aurora]

I've encountered the same issue, and just assumed that telemetry metrics are only emitted from the active node. Looking at the linked discussion, it seems that is not the case.

[michelvocks]

Hi @tstoermer!

Thanks for opening this issue. This is indeed a bug and we came up with #8280 which not necessarily fixes the bug but at least returns the correct error message back to the client.

By default, standby nodes don't have an active token store which is required to validate the provided token from the client. This makes it currently impossible for us to allow sys/metrics requests on standby nodes. However, this only applies to authenticated metrics access and is not the case for unauthenticated metrics access. That means, sys/metrics requests on standby nodes where unauthenticated metrics access has been enabled should work as expected.

Cheers,
Michel

[michelvocks]

Hi @tstoermer!

I'm going to close this since we haven't heard from you for a while.
Feel free to open a new issue if there is anything else you would like to discuss regarding this topic.

Cheers,
Michel