CoreDNS ConfigMap patch - upstream keyword in EKS with CoreDNS 1.7.0

[lescactus]

Bug Report

When deploying the latest version of Maesh with Helm on a AWS EKS cluster (1.18), the controller is still patching CoreDNS Corefile with the upstream keyword, despite CoreDNS version > 1.7

The CoreDNS image deployed by EKS is 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1

What did you do?

Installed Maesh v1.4.1 with the official Helm chart traefik-mesh-3.0.6 on a fresh AWS EKS cluster version v1.18

What did you expect to see?

The controller not patching the CoreDNS ConfigMap with the upstream keyword since the CoreDNS version is >1.7 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1

What did you see instead?

The CoreDNS ConfigMap being patched by the controller init container with the upstream keyword, thus leading to CoreDNS crashing with the following error message: Error during parsing: unknown property 'upstream'

Output of controller log: (What version of Traefik Mesh are you using?)

Maesh version v1.4.1

$ kubectl logs -c traefik-mesh-prepare traefik-mesh-controller-54b79ddc9f-47h6v
{"level":"debug","msg":"Starting prepare...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Using masterURL: \"\"","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Using kubeconfig: \"\"","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Creating in-cluster client","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Building Kubernetes Client...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Building SMI Access Client...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Building SMI Specs Client...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Building SMI Split Client...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"ACL mode enabled: false","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Detecting DNS provider...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Checking if CoreDNS is installed in namespace \"kube-system\"...","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"CoreDNS \"1.7.0-eksbuild.1\" has been detected","time":"2021-02-09T18:56:34Z"}
{"level":"debug","msg":"Patching ConfigMap \"coredns\" in namespace \"kube-system\"...","time":"2021-02-09T18:56:34Z"}
{"level":"info","msg":"CoreDNS ConfigMap \"coredns\" in namespace \"kube-system\" has successfully been patched","time":"2021-02-09T18:56:34Z"}
{"level":"info","msg":"Restarting \"coredns\" pods","time":"2021-02-09T18:56:34Z"}

What is your environment & configuration (arguments, provider, platform, ...)?

AWS EKS v1.18
CoreDNS v1.7.0-eksbuild.1

If applicable, please paste the yaml objects required to reproduce your issue

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"eks.amazonaws.com/component":"coredns","k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS"},"name":"coredns","namespace":"kube-system"},"spec":{"replicas":2,"selector":{"matchLabels":{"eks.amazonaws.com/component":"coredns","k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"annotations":{"eks.amazonaws.com/compute-type":"ec2"},"labels":{"eks.amazonaws.com/component":"coredns","k8s-app":"kube-dns"}},"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/os","operator":"In","values":["linux"]},{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64","arm64"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"k8s-app","operator":"In","values":["kube-dns"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"602401143452.dkr.ecr.ap-south-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":5,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"successThreshold":1,"timeoutSeconds":5},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"}},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume","readOnly":true},{"mountPath":"/tmp","name":"tmp"}]}],"dnsPolicy":"Default","priorityClassName":"system-cluster-critical","serviceAccountName":"coredns","tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"},{"key":"CriticalAddonsOnly","operator":"Exists"}],"volumes":[{"emptyDir":{},"name":"tmp"},{"configMap":{"items":[{"key":"Corefile","path":"Corefile"}],"name":"coredns"},"name":"config-volume"}]}}}}
  creationTimestamp: "2021-02-09T17:56:29Z"
  generation: 4
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
    kubernetes.io/name: CoreDNS
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:eks.amazonaws.com/component: {}
          f:k8s-app: {}
          f:kubernetes.io/name: {}
      f:spec:
        f:progressDeadlineSeconds: {}
        f:replicas: {}
        f:revisionHistoryLimit: {}
        f:selector:
          f:matchLabels:
            .: {}
            f:eks.amazonaws.com/component: {}
            f:k8s-app: {}
        f:strategy:
          f:rollingUpdate:
            .: {}
            f:maxSurge: {}
            f:maxUnavailable: {}
          f:type: {}
        f:template:
          f:metadata:
            f:annotations:
              .: {}
              f:eks.amazonaws.com/compute-type: {}
            f:labels:
              .: {}
              f:eks.amazonaws.com/component: {}
              f:k8s-app: {}
          f:spec:
            f:affinity:
              .: {}
              f:nodeAffinity:
                .: {}
                f:requiredDuringSchedulingIgnoredDuringExecution:
                  .: {}
                  f:nodeSelectorTerms: {}
              f:podAntiAffinity:
                .: {}
                f:preferredDuringSchedulingIgnoredDuringExecution: {}
            f:containers:
              k:{"name":"coredns"}:
                .: {}
                f:args: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:livenessProbe:
                  .: {}
                  f:failureThreshold: {}
                  f:httpGet:
                    .: {}
                    f:path: {}
                    f:port: {}
                    f:scheme: {}
                  f:initialDelaySeconds: {}
                  f:periodSeconds: {}
                  f:successThreshold: {}
                  f:timeoutSeconds: {}
                f:name: {}
                f:ports:
                  .: {}
                  k:{"containerPort":53,"protocol":"TCP"}:
                    .: {}
                    f:containerPort: {}
                    f:name: {}
                    f:protocol: {}
                  k:{"containerPort":53,"protocol":"UDP"}:
                    .: {}
                    f:containerPort: {}
                    f:name: {}
                    f:protocol: {}
                  k:{"containerPort":9153,"protocol":"TCP"}:
                    .: {}
                    f:containerPort: {}
                    f:name: {}
                    f:protocol: {}
                f:readinessProbe:
                  .: {}
                  f:failureThreshold: {}
                  f:httpGet:
                    .: {}
                    f:path: {}
                    f:port: {}
                    f:scheme: {}
                  f:periodSeconds: {}
                  f:successThreshold: {}
                  f:timeoutSeconds: {}
                f:resources:
                  .: {}
                  f:limits:
                    .: {}
                    f:memory: {}
                  f:requests:
                    .: {}
                    f:cpu: {}
                    f:memory: {}
                f:securityContext:
                  .: {}
                  f:allowPrivilegeEscalation: {}
                  f:capabilities:
                    .: {}
                    f:add: {}
                    f:drop: {}
                  f:readOnlyRootFilesystem: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:volumeMounts:
                  .: {}
                  k:{"mountPath":"/etc/coredns"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                    f:readOnly: {}
                  k:{"mountPath":"/tmp"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
            f:dnsPolicy: {}
            f:priorityClassName: {}
            f:restartPolicy: {}
            f:schedulerName: {}
            f:securityContext: {}
            f:serviceAccount: {}
            f:serviceAccountName: {}
            f:terminationGracePeriodSeconds: {}
            f:tolerations: {}
            f:volumes:
              .: {}
              k:{"name":"config-volume"}:
                .: {}
                f:configMap:
                  .: {}
                  f:defaultMode: {}
                  f:items: {}
                  f:name: {}
                f:name: {}
              k:{"name":"tmp"}:
                .: {}
                f:emptyDir: {}
                f:name: {}
    manager: kubectl
    operation: Update
    time: "2021-02-09T17:56:29Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:deployment.kubernetes.io/revision: {}
      f:status:
        f:availableReplicas: {}
        f:conditions:
          .: {}
          k:{"type":"Available"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Progressing"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:observedGeneration: {}
        f:readyReplicas: {}
        f:replicas: {}
        f:unavailableReplicas: {}
        f:updatedReplicas: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-02-09T18:56:34Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:template:
          f:metadata:
            f:annotations:
              f:traefik-mesh-hash: {}
    manager: traefik-mesh
    operation: Update
    time: "2021-02-09T18:56:34Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "13656"
  selfLink: /apis/apps/v1/namespaces/kube-system/deployments/coredns
  uid: 59d0a317-6cbd-4325-96f5-bf47c2327b59
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      eks.amazonaws.com/component: coredns
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        eks.amazonaws.com/compute-type: ec2
        traefik-mesh-hash: 813eef77-c814-4448-912c-2cc050fd86d1
      creationTimestamp: null
      labels:
        eks.amazonaws.com/component: coredns
        k8s-app: kube-dns
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: In
                values:
                - linux
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app
                  operator: In
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: 602401143452.dkr.ecr.ap-south-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
        - mountPath: /tmp
          name: tmp
      dnsPolicy: Default
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: coredns
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - emptyDir: {}
        name: tmp
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume
---
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }



    #### Begin Maesh Block
    maesh:53 {
        errors
        rewrite continue {
            name regex ([a-zA-Z0-9-_]*)\.([a-zv0-9-_]*)\.maesh maesh-{1}-6d61657368-{2}.maesh.svc.cluster.local
            answer name maesh-([a-zA-Z0-9-_]*)-6d61657368-([a-zA-Z0-9-_]*)\.maesh\.svc\.cluster\.local {1}.{2}.maesh
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    #### End Maesh Block

    #### Begin Traefik Mesh Block
    traefik.mesh:53 {
        errors
        rewrite continue {
            name regex ([a-zA-Z0-9-_]*)\.([a-zv0-9-_]*)\.traefik.mesh maesh-{1}-6d61657368-{2}.maesh.svc.cluster.local
            answer name maesh-([a-zA-Z0-9-_]*)-6d61657368-([a-zA-Z0-9-_]*)\.maesh\.svc\.cluster\.local {1}.{2}.traefik.mesh
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    #### End Traefik Mesh Block
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"eks.amazonaws.com/component":"coredns","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2021-02-09T17:56:29Z"
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:eks.amazonaws.com/component: {}
          f:k8s-app: {}
    manager: kubectl
    operation: Update
    time: "2021-02-09T17:56:29Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:Corefile: {}
    manager: traefik-mesh
    operation: Update
    time: "2021-02-09T18:56:34Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "13620"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 6cec9ad9-6a92-4d13-a978-083e39484f66

[renatomjr]

Same here. This makes it impossible to use the Traefik Mesh in eks

[0rax]

I think I just found the issue which was "introduced" by previous PR #774 when I added support for build version with suffixes. It seems like goversion considers v1.7.0-eksbuild.1 as a pre-release version of v1.7.0 while it is not.

versionCoreDNS17 := goversion.Must(goversion.NewVersion("v1.7"))
coreDNSVersion := goversion.Must(goversion.NewVersion("v1.7.0-eksbuild.1"))
fmt.Println(coreDNSVersion.LessThan(versionCoreDNS17)) // Prints true

I will submit a patch to make sure this is fixed while I am looking at adding support for CoreDNS 1.8.

---

EDIT: It seems like there is two solutions:

• Either compare the current CoreDNS version with the last version to support the upstream keyword:

coreDNSVersion := goversion.Must(goversion.NewVersion("v1.7.0-eksbuild.1"))
coreDNSVersion.LessThanOrEqualTo(goversion.Must(goversion.NewVersion("1.6.9"))

• Or strip the pre-release part of the version using version.Core() which would require an update to the go-version package:

coreDNSVersion := goversion.Must(goversion.NewVersion("v1.7.0-eksbuild.1"))
coreDNSVersion.Core().LessThan(goversion.Must(goversion.NewVersion("1.7"))

This issue seems to be something to consider more broadly during the version check in prepare as it might consider v1.8.0-eksbuild.0 as a pre-release and satisfy the LessThan check for the build version.

[kevinpollet]

Closed by #790