-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add loki canary to helm chart #7019
Labels
Comments
Hell yes! |
@dannykopping would you have the alerts for the Canary? |
This is what we use internally at Grafana Labs. {
name: 'canaries_alerts',
rules: [
{
// End to end latency as seen by our canaries.
alert: 'LokiCanaryLatency',
expr: |||
histogram_quantile(0.99, sum(rate(loki_canary_response_latency_seconds_bucket[5m])) by (le, namespace, cluster)) > 5
|||,
'for': '15m',
labels: {
severity: 'warning',
},
annotations: {
message: |||
{{ $labels.job }} is experiencing {{ printf "%.2f" $value }}s 99th percentile latency.
|||,
},
},
],
}, |
4 tasks
trevorwhitney
added a commit
that referenced
this issue
Sep 21, 2022
**What this PR does / why we need it**: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. **Which issue(s) this PR fixes**: Fixes #7019 Co-authored-by: Karsten Jeschkies <[email protected]> Co-authored-by: Dylan Guedes <[email protected]>
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this issue
Nov 7, 2022
**What this PR does / why we need it**: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. **Which issue(s) this PR fixes**: Fixes grafana#7019 Co-authored-by: Karsten Jeschkies <[email protected]> Co-authored-by: Dylan Guedes <[email protected]>
mraboosk
pushed a commit
to mraboosk/loki
that referenced
this issue
Oct 7, 2024
**What this PR does / why we need it**: This PR adds the Loki canary as well as the GEL provisioner to the Loki helm chart. The Loki canary expands the self-monitoring functionality of the Helm chart, giving operators insight beyond whether Loki is just running, but also if it's functioning properly. The GEL provisioner was necessary for the canary to work when the enterprise option (`enterprise.enabled = true`) is enabled. Since enabling enterprise in the helm chart also enables auth/multi-tenancy, the addition of the provisioner allows the helm chart to automatically create a tenant for the canary via the GEL admin API, and to create applicable access policies and tokens. Furthermore, additional tenants can also be specified, which will be provisioned, creating a read and write token for each that will be stored in k8s secrets. **Which issue(s) this PR fixes**: Fixes grafana#7019 Co-authored-by: Karsten Jeschkies <[email protected]> Co-authored-by: Dylan Guedes <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
When I install Loki via the helm chart, how do I know it's working?
Describe the solution you'd like
Include the loki-canary helm chart with the loki helm chart.
Describe alternatives you've considered
Document installing both charts.
Additional context
Adding the canary allows us to add a test that uses it as well.
The text was updated successfully, but these errors were encountered: