[stable/prometheus] support scraping containers by default

[pohly]

Is your feature request related to a problem? Please describe.

As described in prometheus/prometheus#3756, it is sometimes useful to scrape not just one port per pod, but one port per container.

The upstream discussion has been going on for a while and it is not clear how it will be solved. Upstream has disabled the example config for pods (the one with the prometheus.io/scrape annotation), but the Helm chart still has it and documents it as the way to get pods discovered on Kubernetes.

Describe the solution you'd like

A possible solution would be to scrape each container which has a container port with a fixed name and where the pod has suitable annotations. Here's an implementation of that approach:

extraScrapeConfigs: |
  # Example scrape config for containers inside a pod.
  #
  # The relabeling allows the actual container scrape endpoints to be configured via the
  # following annotations:
  #
  # * `prometheus.io/scrape`: Only scrape containers in pods that have a value of `containers`
  # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. The path
  #   has to be the same for all containers because it can only be set per-pod (containers
  #   and container ports have no annotations).
  #
  # The port for each container is specified with a container port that must have the name "metrics".

  - job_name: 'kubernetes-containers'

    kubernetes_sd_configs:
      - role: pod

    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: containers
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: metrics
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_container_port_number]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: kubernetes_pod_container_name
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: kubernetes_pod_node_name # This is useful for daemon sets where pods typically have no meaningful name.
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

This solution is limited to using the same path for all containers, but IMHO that is acceptable.

As it stands above, one can add that job with -f:

helm install prometheus stable/prometheus --set alertmanager.persistentVolume.enabled=false,server.persistentVolume.enabled=false -f kubernetes-container-ports.yaml

I think it would be useful to add it to the default values.yaml, including a "slow" variant.

Describe alternatives you've considered

I tried the approach of merging metrics data with https://github.com/rebuy-de/exporter-merger, but ran into issues when different containers report the same metrics.

Users could be instructed to reconfigure Prometheus manually instead of changing the defaults, but given that nothing in the job above is specific to an application, that seems sub-optimal.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.