Track node and service counts in the state store and emit them periodically as metrics #8603
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We update the usage table on Commit() by using the TrackedChanges() API of memdb. Track memdb changes on restore so that usage data can be compiled
Using the newly provided state store methods, we periodically emit usage metrics from the servers. We decided to emit these metrics from all servers, not just the leader, because that means we do not have to care about leader election flapping causing metrics turbulence, and it seems reasonable for each server to emit its own view of the state, even if they should always converge rapidly.
We add a WriteTxn interface for use in updating the usage memdb table, with the forward-looking prospect of incrementally converting other functions to accept interfaces. As well, we use the ReadTxn in new usage code, and as a side effect convert a couple of existing functions to use that interface as well.
This commit refactors the state store usage code to track unique service name changes on transaction commit. This means we only need to lookup usage entries when reading the information, as opposed to iterating over a large number of service indices. - Take into account a service instance's name being changed - Do not iterate through entire list of service instances, we only care about whether there is 0, 1, or more than 1.
This is below the 10 second interval that lib/telemetry.go implements as its aggregation interval, ensuring that we always report these metrics.
crhino
force-pushed
the
feature/usage-metrics
branch
from
06d6c89 to
40cbd5a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OSS PR usage metrics.
This PR emits 3 new metrics, consul.state.nodes/consul.state.services/consul.state.service_instances, which are all gauges that track the number of registered entities in the Consul state store.
As part of this PR, we add functionality to the state store to keep track of the count of elements associated to a table. This is done by using the change tracking functionality of memdb and a new usage table. This makes it so that we do not need to iterate through the entire table just to retrieve a count of the number of nodes/service instances.
The new functionality is located in agent/consul/state/usage*.go and agent/consul/usagemetrics/usagemetrics*.go.
In addition, a number of state store methods were modified to accept interfaces, either ReadTxn or WriteTxn depending on what each function requires.