cadvisor:latest still looks for inotify, which breaks it on RHEL7.6

[ElCoyote27]

Hi,
I know there have been numerous changes related to Cadvisor with CentOS/RHEL but even though cAdvisor is supposed to use fsnotify, I'm still getting a failure when starting latest:

$ docker run --rm --name cadvisor --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:ro -v /sys:/sys:ro -p 8080:8080 -eSERVICE_TAGS=prom_monitored google/cadvisor:latest

W0128 21:59:43.148299 1 manager.go:349] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory

W0128 21:59:43.187028 1 container.go:409] Failed to create summary reader for "/system.slice/system-openvpn\\x2dserver.slice/[email protected]": none of the resources are being tracked.

F0128 21:59:43.417116 1 cadvisor.go:172] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

Versions 0.27.4 and 0.28.0 work but version 0.28.1 does not.

I cannot really use the --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu trick because this collides with with the volume mount of '/sys'.
I cannot use the ln -s .. trick on the host filesystem because it modifies the underlying Linux OS.

Is there a way to make Go engine inside of the container work with both RHEL and Ubuntu paths?

[dashpole]

hmmm... We really need to move to fsnotify, but I haven't had any bandwidth. Ill see if I can prioritize it, but can't make any guarantees.

[dashpole]

See #1708

[daverodgers77]

See #1708

is there any update on this? i need to use cadvisor running rhel7.8 but as yet i cant find a working docker run command that doesnt throw an error.

many thanks.

[daverodgers77]

so just an update form my side. i managed to get the container running using this set cfg:

docker run
--volume=/:/rootfs:ro
--volume=/var/run:/var/run:rw
--volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu
--volume=/var/lib/docker/:/var/lib/docker:ro
--publish=8080:8080
--detach=true
--name=cadvisor
--privileged=true
google/cadvisor:latest

even though i have removed the --volume=/sys:/sys:ro \ line , i still seem to be getting container stats, which some people said they didnt get if they removed this line. im not sure why it seems to work for me.

[adityai]

so just an update form my side. i managed to get the container running using this set cfg:

docker run
--volume=/:/rootfs:ro
--volume=/var/run:/var/run:rw
--volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu
--volume=/var/lib/docker/:/var/lib/docker:ro
--publish=8080:8080
--detach=true
--name=cadvisor
--privileged=true
google/cadvisor:latest

even though i have removed the --volume=/sys:/sys:ro \ line , i still seem to be getting container stats, which some people said they didnt get if they removed this line. im not sure why it seems to work for me.

This worked for me on RHEL7

[theten52]

this is my system version:

$ lsb_release -a                                                                                                                                                                                              
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.9.2009 (Core)
Release:	7.9.2009
Codename:	Core

this is my docker info:

└─(23:40:27)──> docker info                                                                                                                                                                                                 
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 8891bca22c049cd2dcf13ba2438c0bac8d7f3343 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-1160.45.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 7.717 GiB
Name: wj-c7-01
ID: KH7W:V4Y6:CR36:OSJK:QPLO:RRJ3:GTOJ:5A57:QREM:KHAT:5VGA:3W56
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://192.168.0.106:7890/
Https Proxy: http://192.168.0.106:7890/
No Proxy: localhost,127.0.0.1,.example.com
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

and I use this command worked:

VERSION=v0.45.0 # use the latest release version from https://github.com/google/cadvisor/releases
sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=9101:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:$VERSION