OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host #640

MaxRink · 2023-03-10T11:06:46Z

When setting the OOMWatch featuregate the helm-controller doesnt start anymore.

{"level":"info","ts":"2023-03-10T10:56:07.290Z","logger":"setup","msg":"loading feature gate","DetectDrift":true}
{"level":"info","ts":"2023-03-10T10:56:07.290Z","logger":"setup","msg":"loading feature gate","OOMWatch":true}
{"level":"info","ts":"2023-03-10T10:56:07.290Z","logger":"setup","msg":"loading feature gate","AllowDNSLookups":true}
{"level":"info","ts":"2023-03-10T10:56:07.334Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2023-03-10T10:56:07.335Z","logger":"setup","msg":"setting up OOM watcher"}
{"level":"error","ts":"2023-03-10T10:56:07.335Z","logger":"setup","msg":"unable to setup OOM watcher","error":"failed to stat memory.current \"/sys/fs/cgroup/memory.current\": lstat /sys/fs/cgroup/memory.current: no such file or directory","stacktrace":"main.main\n\t./main.go:215\nruntime.main\n\truntime/proc.go:250"}

Version: v0.31.0
Distro: K8s 1.23.16, on ubuntu 2004 with containerd 1.6.18

Manifest

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    cni.projectcalico.org/containerID: e50928c9ef7107ea6bca2d1371416827e89d3dc70fd5a1b6028eec51c42b34ff
    cni.projectcalico.org/podIP: 192.168.53.173/32
    cni.projectcalico.org/podIPs: 192.168.53.173/32
    container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2023-03-10T10:50:03Z"
  generateName: helm-controller-7bd4f765b7-
  labels:
    app: helm-controller
    das-schiff.telekom.de/owner: das-schiff
    owner: das-schiff
    pod-template-hash: 7bd4f765b7
    schiff.telekom.de/owner: das-schiff
  name: helm-controller-7bd4f765b7-t6hjb
  namespace: flux-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: helm-controller-7bd4f765b7
    uid: 4c3b09d7-f591-4ecd-964d-2436e224be9d
  resourceVersion: "459000519"
  uid: b2be67cb-a6d3-434f-8bcf-5203773a9db1
spec:
  containers:
  - args:
    - --events-addr=http://notification-controller.flux-system.svc.cluster.local./
    - --watch-all-namespaces
    - --log-level=debug
    - --log-encoding=json
    - --enable-leader-election
    - --feature-gates=DetectDrift=true,OOMWatch=true,AllowDNSLookups=true
    env:
    - name: RUNTIME_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: ghcr.io/fluxcd/helm-controller:v0.31.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: healthz
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: manager
    ports:
    - containerPort: 8080
      name: http-prom
      protocol: TCP
    - containerPort: 9440
      name: healthz
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readyz
        port: healthz
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 64Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: temp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-74cgf
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: schiff-cp-swe-md-0-67fdf5d647-5gml7
  preemptionPolicy: PreemptLowerPriority
  priority: 2000000000
  priorityClassName: system-cluster-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1337
    runAsUser: 10000
  serviceAccount: helm-controller
  serviceAccountName: helm-controller
  terminationGracePeriodSeconds: 600
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: temp
  - name: kube-api-access-74cgf
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:50:01Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:50:01Z"
    message: 'containers with unready status: [manager]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:50:01Z"
    message: 'containers with unready status: [manager]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:50:12Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://3264a0b2218ce3559eb083e63a0bce360923855f73c8dba132ec176652720e4b
    image: ghcr.io/fluxcd/helm-controller:v0.31.0
    imageID: ghcr.io/fluxcd/helm-controller@sha256:4c90f62a924057a67e992b7babafe4e47fb52ca5ad772b6a93835986c8441836
    lastState:
      terminated:
        containerID: containerd://3264a0b2218ce3559eb083e63a0bce360923855f73c8dba132ec176652720e4b
        exitCode: 1
        finishedAt: "2023-03-10T11:06:22Z"
        reason: Error
        startedAt: "2023-03-10T11:06:22Z"
    name: manager
    ready: false
    restartCount: 8
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=manager pod=helm-controller-7bd4f765b7-t6hjb_flux-system(b2be67cb-a6d3-434f-8bcf-5203773a9db1)
        reason: CrashLoopBackOff
  hostIP: 10.32.236.140
  phase: Running
  podIP: 192.168.53.173
  podIPs:
  - ip: 192.168.53.173
  qosClass: Burstable
  startTime: "2023-03-10T10:50:01Z"

The text was updated successfully, but these errors were encountered:

hiddeco · 2023-03-10T11:18:02Z

Can you go into the container without the feature enabled and report back at what path your cgroup files are located? We probably need to either make the path configurable or do autodiscovery based on well known paths.

MaxRink · 2023-03-10T11:24:57Z

Sure

ls /sys/fs/cgroup/memory/
cgroup.clone_children               memory.kmem.limit_in_bytes          memory.kmem.tcp.usage_in_bytes      memory.oom_control                  memory.use_hierarchy
cgroup.event_control                memory.kmem.max_usage_in_bytes      memory.kmem.usage_in_bytes          memory.pressure_level               notify_on_release
cgroup.procs                        memory.kmem.slabinfo                memory.limit_in_bytes               memory.soft_limit_in_bytes          tasks
memory.failcnt                      memory.kmem.tcp.failcnt             memory.max_usage_in_bytes           memory.stat
memory.force_empty                  memory.kmem.tcp.limit_in_bytes      memory.move_charge_at_immigrate     memory.swappiness
memory.kmem.failcnt                 memory.kmem.tcp.max_usage_in_bytes  memory.numa_stat                    memory.usage_in_bytes

hiddeco · 2023-03-10T11:31:09Z

Based on the above, it appears you're still on cgroup v1. I will make it configurable by flag, while also adding auto-discovery for well known cgroup v1/v2 paths.

Can you please confirm memory.usage_in_bytes reports the current memory usage, and memory.limit_in_bytes reports the configured memory limit?

MaxRink · 2023-03-10T11:33:27Z

Yes, that seems to the the case

/ $ cat /sys/fs/cgroup/memory/memory.usage_in_bytes 
293511168
/ $ cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
1073741824
k --context schiff-cp-swe.devswe.bn -n flux-system top pod
NAME                                                              CPU(cores)   MEMORY(bytes)   
helm-controller-74c68b45d-x7585                                   85m          260Mi

hiddeco · 2023-03-10T11:34:10Z

Awesome, I'll have something which works for your setup as well later today. Sorry for any inconvenience this may have caused.

hiddeco · 2023-03-10T13:28:29Z

The following RC ghcr.io/fluxcd/helm-controller:rc-65415243 build by CI from #641 should work for you.

See https://fluxcd.io/flux/cheatsheets/bootstrap/#test-release-candidates for information about using release candidates.

hiddeco · 2023-03-10T14:09:45Z

If you are able to run the RC, it would be great if you can report back about it working so I can release the PR with a peace of mind. Thank you 🙇

charlie-haley · 2023-03-10T14:14:13Z

I was encountering the same issue and just tested the RC tag and it's all working as expected on my end

➜ kubectl logs helm-controller-597587f457-pjlsv -f
{"level":"info","ts":"2023-03-10T14:12:10.272Z","logger":"setup","msg":"loading feature gate","OOMWatch":true}
{"level":"info","ts":"2023-03-10T14:12:10.670Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2023-03-10T14:12:10.671Z","logger":"setup","msg":"setting up OOM watcher"}
{"level":"info","ts":"2023-03-10T14:12:10.671Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-03-10T14:12:10.671Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}

hiddeco · 2023-03-10T15:30:37Z

Now available in official Flux v0.41.1 release.

charlie-haley · 2023-03-10T15:32:42Z

Thanks for the quick fix @hiddeco!

hiddeco added the bug Something isn't working label Mar 10, 2023

hiddeco changed the title ~~OOMWatch featuregate leads to not starting pod~~ OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host Mar 10, 2023

hiddeco mentioned this issue Mar 10, 2023

oomwatch: auto-detect well known cgroup paths #641

Merged

hiddeco closed this as completed in #641 Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host #640

OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host #640

MaxRink commented Mar 10, 2023 •

edited

Loading

hiddeco commented Mar 10, 2023

MaxRink commented Mar 10, 2023

hiddeco commented Mar 10, 2023

MaxRink commented Mar 10, 2023

hiddeco commented Mar 10, 2023

hiddeco commented Mar 10, 2023

hiddeco commented Mar 10, 2023

charlie-haley commented Mar 10, 2023

hiddeco commented Mar 10, 2023

charlie-haley commented Mar 10, 2023

OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host #640

OOMWatch feature gate leads to not starting pod on Ubuntu 20.04 host #640

Comments

MaxRink commented Mar 10, 2023 • edited Loading

hiddeco commented Mar 10, 2023

MaxRink commented Mar 10, 2023

hiddeco commented Mar 10, 2023

MaxRink commented Mar 10, 2023

hiddeco commented Mar 10, 2023

hiddeco commented Mar 10, 2023

hiddeco commented Mar 10, 2023

charlie-haley commented Mar 10, 2023

hiddeco commented Mar 10, 2023

charlie-haley commented Mar 10, 2023

MaxRink commented Mar 10, 2023 •

edited

Loading