Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add loki canary to helm chart #7019

Closed
trevorwhitney opened this issue Aug 31, 2022 · 3 comments · Fixed by #7173
Closed

Add loki canary to helm chart #7019

trevorwhitney opened this issue Aug 31, 2022 · 3 comments · Fixed by #7173
Assignees

Comments

@trevorwhitney
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
When I install Loki via the helm chart, how do I know it's working?

Describe the solution you'd like
Include the loki-canary helm chart with the loki helm chart.

Describe alternatives you've considered
Document installing both charts.

Additional context
Adding the canary allows us to add a test that uses it as well.

@dannykopping
Copy link
Contributor

Hell yes!

@jeschkies
Copy link
Contributor

@dannykopping would you have the alerts for the Canary?

@dannykopping
Copy link
Contributor

This is what we use internally at Grafana Labs.

{
  name: 'canaries_alerts',
  rules: [
    {
      // End to end latency as seen by our canaries.
      alert: 'LokiCanaryLatency',
      expr: |||
        histogram_quantile(0.99, sum(rate(loki_canary_response_latency_seconds_bucket[5m])) by (le, namespace, cluster)) > 5
      |||,
      'for': '15m',
      labels: {
        severity: 'warning',
      },
      annotations: {
        message: |||
          {{ $labels.job }} is experiencing {{ printf "%.2f" $value }}s 99th percentile latency.
        |||,
      },
    },
  ],
},

@trevorwhitney trevorwhitney self-assigned this Sep 6, 2022
trevorwhitney added a commit that referenced this issue Sep 21, 2022
**What this PR does / why we need it**:

This PR adds the Loki canary as well as the GEL provisioner to the Loki
helm chart. The Loki canary expands the self-monitoring functionality of
the Helm chart, giving operators insight beyond whether Loki is just
running, but also if it's functioning properly.

The GEL provisioner was necessary for the canary to work when the
enterprise option (`enterprise.enabled = true`) is enabled. Since
enabling enterprise in the helm chart also enables auth/multi-tenancy,
the addition of the provisioner allows the helm chart to automatically
create a tenant for the canary via the GEL admin API, and to create
applicable access policies and tokens. Furthermore, additional tenants
can also be specified, which will be provisioned, creating a read and
write token for each that will be stored in k8s secrets.

**Which issue(s) this PR fixes**:
Fixes #7019 

Co-authored-by: Karsten Jeschkies <[email protected]>
Co-authored-by: Dylan Guedes <[email protected]>
lxwzy pushed a commit to lxwzy/loki that referenced this issue Nov 7, 2022
**What this PR does / why we need it**:

This PR adds the Loki canary as well as the GEL provisioner to the Loki
helm chart. The Loki canary expands the self-monitoring functionality of
the Helm chart, giving operators insight beyond whether Loki is just
running, but also if it's functioning properly.

The GEL provisioner was necessary for the canary to work when the
enterprise option (`enterprise.enabled = true`) is enabled. Since
enabling enterprise in the helm chart also enables auth/multi-tenancy,
the addition of the provisioner allows the helm chart to automatically
create a tenant for the canary via the GEL admin API, and to create
applicable access policies and tokens. Furthermore, additional tenants
can also be specified, which will be provisioned, creating a read and
write token for each that will be stored in k8s secrets.

**Which issue(s) this PR fixes**:
Fixes grafana#7019 

Co-authored-by: Karsten Jeschkies <[email protected]>
Co-authored-by: Dylan Guedes <[email protected]>
mraboosk pushed a commit to mraboosk/loki that referenced this issue Oct 7, 2024
**What this PR does / why we need it**:

This PR adds the Loki canary as well as the GEL provisioner to the Loki
helm chart. The Loki canary expands the self-monitoring functionality of
the Helm chart, giving operators insight beyond whether Loki is just
running, but also if it's functioning properly.

The GEL provisioner was necessary for the canary to work when the
enterprise option (`enterprise.enabled = true`) is enabled. Since
enabling enterprise in the helm chart also enables auth/multi-tenancy,
the addition of the provisioner allows the helm chart to automatically
create a tenant for the canary via the GEL admin API, and to create
applicable access policies and tokens. Furthermore, additional tenants
can also be specified, which will be provisioned, creating a read and
write token for each that will be stored in k8s secrets.

**Which issue(s) this PR fixes**:
Fixes grafana#7019 

Co-authored-by: Karsten Jeschkies <[email protected]>
Co-authored-by: Dylan Guedes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants