cadvisor cannot start

[siuwwong5]

Provide a general summary of the issue in the Title above

If you're describing a bug, tell us what should happen
If you're suggesting a change/improvement, tell us how it should work

Expected Behaviour

Current Behaviour

cadvisor docker container cannot start

logs:
I1227 06:57:44.608799 1 manager.go:1178] Started watching for new ooms in manager
W1227 06:57:44.608845 1 manager.go:313] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
I1227 06:57:44.611828 1 manager.go:329] Starting recovery of all containers
I1227 06:57:44.709836 1 manager.go:334] Recovery completed
F1227 06:57:44.711196 1 cadvisor.go:156] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

Possible Solution

Steps to Reproduce (for bugs)

Provide a link to a live example, or steps to reproduce this bug. Include code to reproduce, if relevant:

Your Environment

Include as many relevant details about the environment you experienced the bug in

• Docker version docker version (e.g. Docker 17.0.05 ):
  docker 1.13.1

• Operating System and version (e.g. Linux, Windows, MacOS):
  linux

[siuwwong5]

I made changes in my docker-compose.yml: 146.222.75.203 is the host IP I deploy the prometheus.

deploy:
  placement:
    constraints:
      - node.hostname == 146.222.75.203

[llitfkitfk]

@siuwwong5 change 146.222.75.203 to "hostname"

[siuwwong5]

@llitfkitfk change 146.222.75.203 to the hostname, right?
"node.hostname == hln2184p", change like this?

[siuwwong5]

@llitfkitfk Thanks a lot! it is working now

[root@hln2184p prometheus-version-2]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2d36cec7771 prom/alertmanager@sha256:959b4671878182f7bc3ecb38bcf05b5f6f1656b4e47c25a5c85c4f6396de68d9 "/bin/alertmanager..." 2 minutes ago Up About a minute 9093/tcp prom_alertmanager.1.fpsta9radkojjpq99ocxjvasu
a3e4e05dac07 prom/prometheus@sha256:ddea229b83e0e3f3a4492ffe57dc9dc239ef8556b659189b09cced52276cda63 "/bin/prometheus -..." 2 minutes ago Up About a minute 9090/tcp prom_prometheus.1.8q7qn1n529ekb12pr14deqtsl
dcab44415906 prom/node-exporter@sha256:8646bb09fa339dce8909d5694713df510b47191e0af5bead70e13fc2315b77af "/bin/node_exporte..." 15 minute s ago Up 15 minutes 9100/tcp prom_node-exporter.z0rnaj9kqjooh8wrp4bofmhpq.ei2mq9i4itnmke42t4iiy8teg
497103bbfe47 grafana/grafana@sha256:f45edaaed369e61a652755c9a8ffbd822daf7b80433b7fb2da49f1d4322cc3d2 "/run.sh" 15 minute s ago Up 15 minutes 3000/tcp prom_grafana.1.tax8olio9p0sg8cnl5bg8he64

[siuwwong5]

@llitfkitfk Sorry, the cadvisor cannot start, and seems it keeps restart.

[llitfkitfk]

@siuwwong5 Can I see the log?

[siuwwong5]

[root@hln2184p prometheus-version-2]# docker logs --follow ca3cb8ab191d
I1228 02:45:16.711905 1 storagedriver.go:50] Caching stats in memory for 2m0s
I1228 02:45:16.712716 1 manager.go:151] cAdvisor running in container: "/sys/fs/cgroup/cpuacct,cpu"
I1228 02:45:16.965053 1 fs.go:139] Filesystem UUIDs: map[]
I1228 02:45:16.965085 1 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/dev major:0 minor:84 fsType:tmpfs blockSize:0} /dev/sdb1:{mountpoint:/rootfs/boot major:8 minor:17 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-1e927d4a262c7001e218f05b04293e51fa3daf201242fd5309c8be16415b1a1b:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/1e927d4a262c7001e218f05b04293e51fa3daf201242fd5309c8be16415b1a1b major:253 minor:7 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-ce45fa2c590220cb5b01fd0b862ea898ab811cd7410badc3a1dc4c6993d759c9:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/ce45fa2c590220cb5b01fd0b862ea898ab811cd7410badc3a1dc4c6993d759c9 major:253 minor:8 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-3aa0ce1e0c8c35f35d559cef545960f553fb4ce8e90dc92ead2a1dfbb8d68d7d:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/3aa0ce1e0c8c35f35d559cef545960f553fb4ce8e90dc92ead2a1dfbb8d68d7d major:253 minor:11 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-e98ede9df3f3e933fd3831441a4302e90cd9c2d8f9a6902c39de33c58c91f1c2:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/e98ede9df3f3e933fd3831441a4302e90cd9c2d8f9a6902c39de33c58c91f1c2 major:253 minor:9 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-629650fdc6d1c17641802c63525ffcf97ed7ac2397f8b5deb86f618501241c10:{mountpoint:/rootfs/var/lib/docker/devicemapper/mnt/629650fdc6d1c17641802c63525ffcf97ed7ac2397f8b5deb86f618501241c10 major:253 minor:10 fsType:xfs blockSize:0} /dev/mapper/docker-253:1-67111133-a17c4a337d613e6c6a0b3c6ac19386c478ccc4823972236289e8334052cc73cc:{mountpoint:/ major:253 minor:12 fsType:xfs blockSize:0} shm:{mountpoint:/rootfs/var/lib/docker/containers/1e927d4a262c7001e218f05b04293e51fa3daf201242fd5309c8be16415b1a1b/shm major:0 minor:40 fsType:tmpfs blockSize:0} /dev/mapper/rhel-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:1 fsType:xfs blockSize:0}]
I1228 02:45:16.971250 1 manager.go:225] Machine: {NumCores:2 CpuFrequency:2599792 MemoryCapacity:4010094592 HugePages:[{PageSize:2048 NumPages:0}] MachineID:ca518a1e76844782ad4ef1da6de5a673 SystemUUID:4239DFA1-2637-B2CB-0996-9A3608496FD4 BootID:cd93d7bb-7243-4d15-99c4-4f302f40ef1f Filesystems:[{Device:/dev/mapper/docker-253:1-67111133-ce45fa2c590220cb5b01fd0b862ea898ab811cd7410badc3a1dc4c6993d759c9 DeviceMajor:253 DeviceMinor:8 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:/dev/mapper/docker-253:1-67111133-3aa0ce1e0c8c35f35d559cef545960f553fb4ce8e90dc92ead2a1dfbb8d68d7d DeviceMajor:253 DeviceMinor:11 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:/dev/mapper/docker-253:1-67111133-e98ede9df3f3e933fd3831441a4302e90cd9c2d8f9a6902c39de33c58c91f1c2 DeviceMajor:253 DeviceMinor:9 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:/dev/mapper/docker-253:1-67111133-629650fdc6d1c17641802c63525ffcf97ed7ac2397f8b5deb86f618501241c10 DeviceMajor:253 DeviceMinor:10 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:/dev/mapper/docker-253:1-67111133-a17c4a337d613e6c6a0b3c6ac19386c478ccc4823972236289e8334052cc73cc DeviceMajor:253 DeviceMinor:12 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:84 Capacity:2005045248 Type:vfs Inodes:489513 HasInodes:true} {Device:/dev/sdb1 DeviceMajor:8 DeviceMinor:17 Capacity:520794112 Type:vfs Inodes:512000 HasInodes:true} {Device:/dev/mapper/docker-253:1-67111133-1e927d4a262c7001e218f05b04293e51fa3daf201242fd5309c8be16415b1a1b DeviceMajor:253 DeviceMinor:7 Capacity:107320705024 Type:vfs Inodes:52428288 HasInodes:true} {Device:/dev/mapper/rhel-root DeviceMajor:253 DeviceMinor:1 Capacity:38195646464 Type:vfs Inodes:29484496 HasInodes:true} {Device:shm DeviceMajor:0 DeviceMinor:40 Capacity:67108864 Type:vfs Inodes:489513 HasInodes:true}] DiskMap:map[8:32:{Name:sdc Major:8 Minor:32 Size:10737418240 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:4160749568 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:38214303744 Scheduler:none} 253:10:{Name:dm-10 Major:253 Minor:10 Size:107374182400 Scheduler:none} 253:11:{Name:dm-11 Major:253 Minor:11 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:210235293696 Scheduler:none} 253:4:{Name:dm-4 Major:253 Minor:4 Size:210235293696 Scheduler:none} 253:6:{Name:dm-6 Major:253 Minor:6 Size:107374182400 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:218103808 Scheduler:none} 253:9:{Name:dm-9 Major:253 Minor:9 Size:107374182400 Scheduler:none} 253:12:{Name:dm-12 Major:253 Minor:12 Size:107374182400 Scheduler:none} 253:5:{Name:dm-5 Major:253 Minor:5 Size:107374182400 Scheduler:none} 253:7:{Name:dm-7 Major:253 Minor:7 Size:107374182400 Scheduler:none} 253:8:{Name:dm-8 Major:253 Minor:8 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:214748364800 Scheduler:deadline} 8:16:{Name:sdb Major:8 Minor:16 Size:42949672960 Scheduler:deadline}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:b9:5e:50 Speed:10000 Mtu:1500} {Name:flannel.1 MacAddress:fe:bc:ad:e4:76:e6 Speed:0 Mtu:1450}] Topology:[{Id:0 Memory:4294500352 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:20971520 Type:Unified Level:3}]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:20971520 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1228 02:45:16.972320 1 manager.go:231] Version: {KernelVersion:3.10.0-514.16.1.el7.x86_64 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.13.1 DockerAPIVersion:1.26 CadvisorVersion:v0.28.3 CadvisorRevision:1e567c2}
E1228 02:45:17.093385 1 factory.go:340] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
I1228 02:45:17.094018 1 factory.go:356] Registering Docker factory
I1228 02:45:19.094405 1 factory.go:54] Registering systemd factory
I1228 02:45:19.096258 1 factory.go:86] Registering Raw factory
I1228 02:45:19.097938 1 manager.go:1178] Started watching for new ooms in manager
W1228 02:45:19.097998 1 manager.go:313] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
I1228 02:45:19.102073 1 manager.go:329] Starting recovery of all containers
I1228 02:45:19.231714 1 manager.go:334] Recovery completed
F1228 02:45:19.293178 1 cadvisor.go:156] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

[siuwwong5]

[root@hln2184p prometheus-version-2]# ls -al /sys/fs/cgroup/cpu*
lrwxrwxrwx. 1 root root 11 May 3 2017 /sys/fs/cgroup/cpu -> cpu,cpuacct
lrwxrwxrwx. 1 root root 11 May 3 2017 /sys/fs/cgroup/cpuacct -> cpu,cpuacct

/sys/fs/cgroup/cpu,cpuacct:
total 0
drwxr-xr-x. 5 root root 0 Dec 4 14:08 .
drwxr-xr-x. 13 root root 340 May 3 2017 ..
-rw-r--r--. 1 root root 0 May 3 2017 cgroup.clone_children
--w--w--w-. 1 root root 0 May 3 2017 cgroup.event_control
-rw-r--r--. 1 root root 0 May 3 2017 cgroup.procs
-r--r--r--. 1 root root 0 May 3 2017 cgroup.sane_behavior
-r--r--r--. 1 root root 0 May 3 2017 cpuacct.stat
-rw-r--r--. 1 root root 0 May 3 2017 cpuacct.usage
-r--r--r--. 1 root root 0 May 3 2017 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 May 3 2017 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 May 3 2017 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 May 3 2017 cpu.rt_period_us
-rw-r--r--. 1 root root 0 May 3 2017 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 May 3 2017 cpu.shares
-r--r--r--. 1 root root 0 May 3 2017 cpu.stat
drwxr-xr-x. 8 root root 0 Dec 28 10:46 docker
-rw-r--r--. 1 root root 0 May 3 2017 notify_on_release
-rw-r--r--. 1 root root 0 May 3 2017 release_agent
drwxr-xr-x. 97 root root 0 Dec 28 04:40 system.slice
-rw-r--r--. 1 root root 0 May 3 2017 tasks
drwxr-xr-x. 2 root root 0 Dec 25 16:06 user.slice

/sys/fs/cgroup/cpuset:
total 0
drwxr-xr-x. 4 root root 0 May 3 2017 .
drwxr-xr-x. 13 root root 340 May 3 2017 ..
-rw-r--r--. 1 root root 0 May 3 2017 cgroup.clone_children
--w--w--w-. 1 root root 0 May 3 2017 cgroup.event_control
-rw-r--r--. 1 root root 0 May 3 2017 cgroup.procs
-r--r--r--. 1 root root 0 May 3 2017 cgroup.sane_behavior
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.cpu_exclusive
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.cpus
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.mem_exclusive
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.mem_hardwall
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.memory_migrate
-r--r--r--. 1 root root 0 May 3 2017 cpuset.memory_pressure
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.memory_pressure_enabled
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.memory_spread_page
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.memory_spread_slab
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.mems
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.sched_load_balance
-rw-r--r--. 1 root root 0 May 3 2017 cpuset.sched_relax_domain_level
drwxr-xr-x. 8 root root 0 Dec 28 10:46 docker
-rw-r--r--. 1 root root 0 May 3 2017 notify_on_release
-rw-r--r--. 1 root root 0 May 3 2017 release_agent
drwxr-xr-x. 2 root root 0 Dec 27 11:25 system.slice
-rw-r--r--. 1 root root 0 May 3 2017 tasks

[llitfkitfk]

google/cadvisor#1846
google/cadvisor#1843

[llitfkitfk]

@siuwwong5 try google/cadvisor#1843 (comment)

[siuwwong5]

Thanks a lot. The fix can help.

[root@hln2184p prometheus-version-2]# mount -o remount,rw '/sys/fs/cgroup'
[root@hln2184p prometheus-version-2]# ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu