support defining metrics_filters for prometheus module in annotations

[newly12]

Please label this PR with one of the following labels, depending on the scope of your change:

• Enhancement

What does this PR do?

support defining metrics_filters for prometheus module in annotations

Why is it important?

support metrics_filters in annotations

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

run test case

Or:

1. Run Metricbeat with hints based autodiscover:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true

2. Deploy a Prometheus sample:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-deployment8
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: prometheus
  replicas: 1 
  template:
    metadata:
      labels:
        app: prometheus
      annotations:
        co.elastic.metrics/module: prometheus
        co.elastic.metrics/hosts: '${data.host}:${data.port}'
        co.elastic.metrics/metrics_path: '/metrics'
        co.elastic.metrics/period: 1m
        co.elastic.metrics/metrics_filters.include: prometheus_http_*
        co.elastic.metrics/metrics_filters.exclude: ^prometheus_http_response_size_bytes$
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus
        ports:
        - containerPort: 9090

3. Verify that filters work properly and only prometheus_http_* metrics are collected while prometheus_http_response_size_bytes are excluded .

Related issues

• Closes parse annotation hint for metrics_filters.<include|exclude> as list for prometheus module #23430

Use cases

annotate pod to add metrics filters without defining all of them in /raw

Screenshots

Logs

2021-03-01T11:09:03.154+0800    DEBUG   [autodiscover.pod]      kubernetes/pod.go:224   Generated builder event map[container:{"id":"3cee712678e78de090ef1f734cede27065dcbd47d8e4393b8fba7801b185ca68","image":"quay.io/freshtracks.io/avalanche","name":"metrics-exporter","runtime":"containerd"} hints:{"metrics":{"hosts":"${data.host}:9001/metrics","metrics_filters":{"exclude":"foo,bar","include":"xxx,yyy"},"module":"prometheus"}} host:10.1.94.211 kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"${data.host}:9001/metrics","metrics/metrics_filters":{"exclude":"foo,bar","include":"xxx,yyy"},"metrics/module":"prometheus"}}},"container":{"id":"3cee712678e78de090ef1f734cede27065dcbd47d8e4393b8fba7801b185ca68","image":"quay.io/freshtracks.io/avalanche","name":"metrics-exporter","runtime":"containerd"},"deployment":{"name":"metrics-exporter"},"labels":{"run":"metrics-exporter"},"namespace":"testing","namespace_uid":"d8707747-d485-492c-8ba6-7746d9a4f0c3","node":{"hostname":"yundeng-ubuntu-3533991","labels":null,"name":"yundeng-ubuntu-3533991","uid":"00828e39-da73-47d7-a826-280e11f5ed04"},"pod":{"name":"metrics-exporter-bc8cfdbc6-f29pr","uid":"cfd69c76-b7b2-4a5d-a7d6-4c715f99ef92"},"replicaset":{"name":"metrics-exporter-bc8cfdbc6"}} port:9001]
2021-03-01T11:09:03.154+0800    DEBUG   [autodiscover]  autodiscover/autodiscover.go:172        Got a start event: map[config:[0xc000995620] host:10.1.94.211 id:cfd69c76-b7b2-4a5d-a7d6-4c715f99ef92 kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"${data.host}:9001/metrics","metrics/metrics_filters":{"exclude":"foo,bar","include":"xxx,yyy"},"metrics/module":"prometheus"}}},"deployment":{"name":"metrics-exporter"},"labels":{"run":"metrics-exporter"},"namespace":"testing","namespace_uid":"d8707747-d485-492c-8ba6-7746d9a4f0c3","node":{"hostname":"yundeng-ubuntu-3533991","labels":null,"name":"yundeng-ubuntu-3533991","uid":"00828e39-da73-47d7-a826-280e11f5ed04"},"pod":{"name":"metrics-exporter-bc8cfdbc6-f29pr","uid":"cfd69c76-b7b2-4a5d-a7d6-4c715f99ef92"},"replicaset":{"name":"metrics-exporter-bc8cfdbc6"}} meta:{"kubernetes":{"deployment":{"name":"metrics-exporter"},"labels":{"run":"metrics-exporter"},"namespace":"testing","namespace_uid":"d8707747-d485-492c-8ba6-7746d9a4f0c3","node":{"hostname":"yundeng-ubuntu-3533991","labels":null,"name":"yundeng-ubuntu-3533991","uid":"00828e39-da73-47d7-a826-280e11f5ed04"},"pod":{"name":"metrics-exporter-bc8cfdbc6-f29pr","uid":"cfd69c76-b7b2-4a5d-a7d6-4c715f99ef92"},"replicaset":{"name":"metrics-exporter-bc8cfdbc6"}}} ports:{"":9001} provider:99997848-4871-4f9e-bbdd-337f6e680a58 start:true]
2021-03-01T11:09:03.154+0800    DEBUG   [registry.lightmodules] mb/lightmodules.go:262  Light modules directory '%!d(string=/private/var/folders/db/rczwwf5j2l553fv9kf_b27j0394k87/T/module)' doesn't exist
2021-03-01T11:09:03.154+0800    DEBUG   [hints.builder] hints/metrics.go:171    generated config: {"enabled":true,"hosts":"xxxxx","metrics_filters":{"exclude":["foo","bar"],"include":["xxx","yyy"]},"metricsets":["collector"],"module":"prometheus","period":"1m","processors":null,"ssl":{},"timeout":"3s"}
2021-03-01T11:09:03.154+0800    DEBUG   [autodiscover]  autodiscover/autodiscover.go:193        Generated config: {
  "enabled": true,
  "hosts": [
    "xxxxx"
  ],
  "metrics_filters": {
    "exclude": [
      "foo",
      "bar"
    ],
    "include": [
      "xxx",
      "yyy"
    ]
  },
  "metricsets": [
    "collector"
  ],
  "module": "prometheus",
  "period": "1m",
  "timeout": "3s"
}

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Started by user Chris Mark

• Start Time: 2021-03-04T11:05:48.157+0000

• Duration: 48 min 53 sec

• Commit: 26017d9

Test stats 🧪

Test	Results
Failed	0
Passed	10144
Skipped	2464
Total	12608

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	10144
Skipped	2464
Total	12608

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ChrsMark]

Looks great, thanks for working on this! I left two minor comments.

Also could you please add the respective part in the docs (https://github.com/elastic/beats/blob/95a88a6ed7f0c175175a1d3772a63c6e8d313437/metricbeat/docs/autodiscover-hints.asciidoc an example too would be nice)?

[ChrsMark]

/test

[newly12]

@ChrsMark thanks for reviewing this PR! PTAL

[ChrsMark]

/test

[newly12]

@ChrsMark there was an error in unit tests. I don't have much insight of that. could you pls help?

[ChrsMark]

@newly12 no worries, it should be an issue with windows node that made the CI failing (it is fixed internally). I'm restarting the job to see if it was fixed.

I also manually tested this one and works perfectly. I put my steps in the PR's description for reference.

Thanks again for working on this!