Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add replication metrics #10073

Merged
merged 2 commits into from
Apr 22, 2021
Merged

Add replication metrics #10073

merged 2 commits into from
Apr 22, 2021

Conversation

mkeeler
Copy link
Member

@mkeeler mkeeler commented Apr 20, 2021 •
edited
Loading

This adds metrics around replication status and latest replicated index to be emitted for:

  • ACL Tokens
  • ACL Policies
  • ACL Roles
  • Legacy ACLs
  • Federation States
  • Config Entries
  • Namespaces

The metrics will be emitted with the name: consul.leader.replication.<type>.status and consul.leader.replication.<type>.index

The status one is 0 if the last round of replication errored and 1 if it didn't. The index one is updated with the last replicated index.

As we don't have unit tests to ensure metrics were emitted correctly I tested this manually like so:

  1. make dev-docker for this branch
  2. git clone github.com/mkeeler/consul-docker-test
  3. cd acls within the consul-docker-test source
  4. terraform init
  5. terraform apply -var consul_image=consul-dev
    • This will bring up a multi-dc setup with ACLs enabled with token replication and connect enabled which is enough to cause all the various replicators to be running.
  6. In a browser go to http://localhost:3000/d/FanxUjuGk/replication?orgId=1
  7. Alternatively navigate to http://localhost:9090 (prometheus ui) and search for the metrics:
    • consul_leader_replication_acl_tokens_status
    • consul_leader_replication_acl_tokens_index
    • consul_leader_replication_acl_policies_status
    • consul_leader_replication_acl_policies_index
    • consul_leader_replication_acl_roles_status
    • consul_leader_replication_acl_roles_index
    • consul_leader_replication_config_entries_status
    • consul_leader_replication_config_entries_index
    • consul_leader_replication_federation_state_status
    • consul_leader_replication_federation_state_index

@vercel vercel bot temporarily deployed to Preview – consul-ui-staging April 22, 2021 14:47 Inactive
@vercel vercel bot temporarily deployed to Preview – consul April 22, 2021 14:47 Inactive
@mkeeler mkeeler marked this pull request as ready for review April 22, 2021 15:07
chapmanc
chapmanc approved these changes Apr 22, 2021
View reviewed changes
Copy link
Contributor

@chapmanc chapmanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mkeeler mkeeler merged commit ecbccdc into master Apr 22, 2021
@mkeeler mkeeler deleted the replication-metrics branch April 22, 2021 15:20
@hashicorp-ci
Copy link
Contributor

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/355347.

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.9.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.8.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/355380.

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.10.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.9.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.8.x failed! Build Log

mkeeler added a commit that referenced this pull request Apr 22, 2021
@mkeeler
Add replication metrics (#10073)
ddb9e20
@hashicorp-ci
Copy link
Contributor

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/356166.

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.10.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.9.x failed! Build Log

@hashicorp-ci
Copy link
Contributor

🍒❌ Cherry pick of commit ecbccdc onto release/1.8.x failed! Build Log

mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler
Add replication metrics (#10073)
6597bf4
@mkeeler mkeeler mentioned this pull request Apr 23, 2021
Merged
@mkeeler mkeeler mentioned this pull request Apr 23, 2021
Merged
mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler
Add replication metrics (#10073)
f880eca
This was referenced Apr 23, 2021
Merged
Merged
mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler @dnephin
[Backport/1.10.x]: Backport #10073 (#10102)
d049590 
* Merge pull request #10094 from hashicorp/update-fingerprint

updating fingerprint

* Add replication metrics (#10073)

Co-authored-by: Daniel Nephin <[email protected]>
mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler @dnephin
[Backport/1.9.x] Backport #10073 (#10104)
7093e48 
* Merge pull request #10094 from hashicorp/update-fingerprint

updating fingerprint

* Add replication metrics (#10073)

Co-authored-by: Daniel Nephin <[email protected]>
mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler
Add replication metrics (#10073)
835d3d4 
# Conflicts:
#	agent/consul/replication.go
mkeeler added a commit that referenced this pull request Apr 23, 2021
@mkeeler
Merge pull request #10103 from hashicorp/backport/1.8.x/10073
86cd4fb 
[Backport/1.8.x] Backport #10073
@mkeeler mkeeler mentioned this pull request Apr 23, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chapmanc chapmanc chapmanc approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mkeeler @hashicorp-ci @chapmanc