cpumanager is not set with static policy

[pperiyasamy]

BUG REPORT

cpumanager is not set with static policy.

I'm trying to create single node k8s cluster with kubeadm init, but it looks like kubelet's cpumanager is not initialized with static policy.

Here is the snippet of kubelet configuration in kubeadm init yaml file:

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  CPUManager: true
  HugePages: true
cpuManagerPolicy: static

Versions

kubeadm version:
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-04-03T04:15:58Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version :
  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-04-03T04:16:27Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-04-03T04:12:37Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"
  CENTOS_MANTISBT_PROJECT="CentOS-7"
  CENTOS_MANTISBT_PROJECT_VERSION="7"
  REDHAT_SUPPORT_PRODUCT="centos"
  REDHAT_SUPPORT_PRODUCT_VERSION="7"
• Kernel (e.g. uname -a):
  Linux my-centos 3.10.0-862.14.4.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others :
  journalctl -u kubelet shows the following

Jun 14 14:18:10 my-centos kubelet[11411]: I0614 14:18:10.618876   11411 cpu_manager.go:155] [cpumanager] starting with none policy
Jun 14 14:18:10 my-centos kubelet[11411]: I0614 14:18:10.618894   11411 cpu_manager.go:156] [cpumanager] reconciling every 10s
Jun 14 14:18:10 my-centos kubelet[11411]: I0614 14:18:10.618914   11411 policy_none.go:42] [cpumanager] none policy: Start

What happened?

pod created with cpu affinity, but all the host's CPU cores are visible inside the pod.

Host:

[root@my-centos ~]# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-7
[root@my-centos ~]# cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive
1

POD:

root@nginx-pod:/# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-7
root@nginx-pod:/# cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive
0

What you expected to happen?

CPU core isolation should happen for the POD for running high performance (example: DPDK) applications.

[neolit123]

my immediate comment, is that i don't think we should enable the "static" policy by default or at least there has to be a really good reason to do so.

if sig-node have plans for the kubelet to switch away from the default affinity (policy=none), kubeadm will follow. until then i think this falls under customizations that the user should apply on demand for certain nodes only.

@kubernetes/sig-cluster-lifecycle

[yagonobre]

I agree with @neolit123, I use static policy in some clusters, but it's more an exception than the normal case of use.

[pperiyasamy]

@neolit123 @yagonobre Thanks for your response. I'm not asking to make cpu manager policy as static by default. I'm trying to set the kubelet with static policy explicitly via kubeadm init yaml file, but it isn't set actually.

I can also see the cpuManagerPolicy set as static in the kubelet-config-1.13 configMap.

[root@my-centos ~]# kubectl describe configMap kubelet-config-1.13 -n kube-system
Name:         kubelet-config-1.13
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
kubelet:
----
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Cache
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
featureGates:
  CPUManager: true
  HugePages: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

Events:  <none>

[neolit123]

cpuManagerPolicy: static is set correctly in the KubeletConfiguration.
so technically this is not a kubeadm bug.

to use "static", make sure your pods get assigned the "Guaranteed" QoS class.
https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/
https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/

[pperiyasamy]

to use "static", make sure your pods get assigned the "Guaranteed" QoS class.

Yes, I can see QoS class set as "Guaranteed" for the Pod as its created with cpu requests/limits both set with 2. But I don't see which are the cpu cores isolated for the Pod so that dpdk application can be run on those specific cores.

root@nginx-pod:/# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-7
root@nginx-pod:/# cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive
0

[neolit123]

you can try the k8s support channels like reddit or stackoverflow:
https://github.com/kubernetes/community/blob/master/contributors/guide/issue-triage.md#user-support-response-example

if you discover a kubelet related bug, please log an issue in kuberentes/kubernetes and tag /sig node.

thanks.