vmagent crashloops with a valid config

[stribb]

Configuration

$ k get --all-namespacesvmpodscrapes.operator.victoriametrics.com -oyaml

apiVersion: v1
items:
- apiVersion: operator.victoriametrics.com/v1beta1
  kind: VMPodScrape
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"operator.victoriametrics.com/v1beta1","kind":"VMPodScrape","metadata":{"annotations":{},"name":"all-pods","namespace":"observability"},"spec":{"namespaceSelector":{"any":true},"podMetricsEndpoints":[{"path":"/mgmt/metrics","port":"monitoring","scheme":"http"},{"relabelConfigs":[{"action":"replace","source_labels":["__meta_kubernetes_pod_annotation_prometheus_io_path"],"target_label":"__metrics_path__"},{"action":"replace","regex":"([^:]+)(?::\\d+)?;(\\d+)","replacement":"$1:$2","source_labels":["__address__","__meta_kubernetes_pod_annotation_prometheus_io_port"],"target_label":"__address__"}]}],"selector":{}}}
    creationTimestamp: "2020-12-17T16:18:32Z"
    generation: 1
    name: all-pods
    namespace: observability
    resourceVersion: "49561733"
    selfLink: /apis/operator.victoriametrics.com/v1beta1/namespaces/observability/vmpodscrapes/all-pods
    uid: a052427e-ea14-4a85-abaf-747a2e89aaf8
  spec:
    namespaceSelector:
      any: true
    podMetricsEndpoints:
    - path: /mgmt/metrics
      port: monitoring
      scheme: http
    - relabelConfigs:
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
    selector: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ k get vmagents.operator.victoriametrics.com -oyaml

apiVersion: v1
items:
- apiVersion: operator.victoriametrics.com/v1beta1
  kind: VMAgent
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"operator.victoriametrics.com/v1beta1","kind":"VMAgent","metadata":{"annotations":{},"name":"vmagent","namespace":"observability"},"spec":{"podScrapeNamespaceSelector":{},"podScrapeSelector":{},"remoteWrite":[{"url":"http://vminsert-vmcluster.observability.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"}],"replicaCount":1,"serviceAccountName":"vmagent","serviceScrapeNamespaceSelector":{},"serviceScrapeSelector":{}}}
    creationTimestamp: "2020-12-14T17:32:30Z"
    generation: 2
    name: vmagent
    namespace: observability
    resourceVersion: "50155052"
    selfLink: /apis/operator.victoriametrics.com/v1beta1/namespaces/observability/vmagents/vmagent
    uid: e45d415c-fb19-410a-8c7e-d4e86a8f0d03
  spec:
    podScrapeNamespaceSelector: {}
    podScrapeSelector: {}
    remoteWrite:
    - url: http://vminsert-vmcluster.observability.svc.cluster.local:8480/insert/0/prometheus/api/v1/write
    replicaCount: 1
    serviceAccountName: vmagent
    serviceScrapeNamespaceSelector: {}
    serviceScrapeSelector: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

vmagent.yaml

# ...
- job_name: observability/all-pods/1
  honor_labels: false
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - target_label: job
    replacement: observability/all-pods
  - action: replace
  - regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    action: replace

Logs

...
2020-12-18T10:21:54.459Z	info	VictoriaMetrics/app/vmagent/main.go:113	started vmagent in 0.001 seconds
2020-12-18T10:21:54.459Z	info	VictoriaMetrics/lib/httpserver/httpserver.go:82	starting http server at http://:8429/
2020-12-18T10:21:54.459Z	info	VictoriaMetrics/lib/httpserver/httpserver.go:83	pprof handlers are exposed at http://:8429/debug/pprof/
2020-12-18T10:21:54.462Z	info	VictoriaMetrics/lib/promscrape/scraper.go:93	reading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
2020-12-18T10:21:54.518Z	fatal	VictoriaMetrics/lib/promscrape/scraper.go:96	cannot read "/etc/vmagent/config_out/vmagent.env.yaml": cannot parse Prometheus config from "/etc/vmagent/config_out/vmagent.env.yaml": cannot parse `scrape_config` #7: cannot parse `relabel_configs` for `job_name` "observability/all-pods/1": error when parsing `relabel_config` #5: missing `target_label` for `action=replace`

Thanks!

[f41gh7]

The problem at relabelConfigs - api has sourceLabels and targetLabel fields instead of source_labels and target_label.

Try to use following config:

  apiVersion: operator.victoriametrics.com/v1beta1
  kind: VMPodScrape
  metadata:
    annotations:
    name: all-pods
    namespace: observability
  spec:
    namespaceSelector:
      any: true
    podMetricsEndpoints:
    - path: /mgmt/metrics
      port: monitoring
      scheme: http
    - relabelConfigs:
      - action: replace
        sourceLabels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        targetLabel: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        # not source_labels
        sourceLabels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        # not target_label
        targetLabel: __address__
    selector: {}

[f41gh7]

Need to fix it with adding target_label and source_labels fields. And replace empty values for targetLabel and sourceLabels with corresponding values.

[f41gh7]

Added patch to master branch, now operator supports relabelConfig sources and target labels in sourceLabels/ targetLabel and classic source_labels / target_label format.

[f41gh7]

included in release https://github.com/VictoriaMetrics/operator/releases/tag/v0.7.0