docs: Add release label to prometheusRule example

[alexandre-allard]

Component: docs

Context:
In the documentation we were missing the release label for PrometheusRules, so rules were not taken into account by Prometheus.

Summary:
Add the label in the example and a note to explain the labels are mandatory.

Acceptance criteria: n/a

---

Closes: #3293

[bert-e]

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• one peer

Peer approvals must include at least 1 approval from the following list:

• @NicolasT

• @ChengYanJin

• @alexandre-allard-scality

• @thomasdanan

• @gdemonet

• @TeddyAndrieux

• @lucieleonard

• @JBWatenbergScality

[alexandre-allard]

/approve

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

• ✔️ development/2.9

The following branches will NOT be impacted:

• development/1.0
• development/1.1
• development/1.2
• development/1.3
• development/2.0
• development/2.1
• development/2.2
• development/2.3
• development/2.4
• development/2.5
• development/2.6
• development/2.7
• development/2.8

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the
  current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/2.9

The following branches have NOT changed:

• development/1.0
• development/1.1
• development/1.2
• development/1.3
• development/2.0
• development/2.1
• development/2.2
• development/2.3
• development/2.4
• development/2.5
• development/2.6
• development/2.7
• development/2.8

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

[NicolasT]

That seems like an issue, no? The app deploys the rule, so app.kubernetes.io/name and app must be related to the app creating the rule, totally unrelated to prometheus-operator. Similarly, release should not be part of any (required) set of labels, and should not be prometheus-release.

In essence, we need to change the set of labels our prometheus-operator uses to filter out rules (and other objects) it should take into account, e.g., turn the selector into metalk8s.scality.com/monitor: "true" or something along those lines.

[alexandre-allard]

Can easily be overridden using prometheus.prometheusSpec.ruleSelector in the charts/kube-prometheus-stack.yaml file:

prometheus:
  prometheusSpec:
    ruleSelector:
      matchLabels:
        metalk8s.scality.com/monitor: "true"

And we should probably do the same for ServiceMonitor.

[gdemonet]

Indeed, that's what we recommend in #2199
This PR was more about making sure our docs are accurate about the current state of the product (although it's debatable whether it's a useful change 😜)

[alexandre-allard]

Oneline fix, does not cost much, at least now it's accurate even if it's ugly (which was already the case).