LXD on OpenRC host not mounting cgroups in containers #4052

palica · 2017-11-24T11:15:50Z

Required information

Distribution: Gentoo
Distribution version: current
The output of "lxc info" or if that fails:

config:        
  core.https_address: 192.168.1.112:9000
  core.https_allowed_credentials: "true"
  core.https_allowed_headers: Origin, X-Requested-With, Content-Type, Accept
  core.https_allowed_methods: GET, POST, PUT, DELETE, OPTIONS
  core.https_allowed_origin: '*'
  core.trust_password: true
api_extensions:           
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki      
- container_last_used_at  
- etag              
- patch                         
- usb_devices                  
- https_allowed_credentials                                         
- image_compression_algorithm                                       
- directory_manipulation                                            
- container_cpu_time                                                
- storage_zfs_use_refquota                                          
- storage_lvm_mount_options                                         
- network                                                           
- profile_usedby                                                    
- container_push                                                    
- container_exec_recording                                          
- certificate_update                                                
- container_exec_signal_handling                                    
- gpu_devices                                                       
- container_image_properties                                        
- migration_progress                                                
- id_map                                                            
- network_firewall_filtering                                        
- network_routes                                                    
- storage                                                           
- file_delete                                                       
- file_append                                                       
- network_dhcp_expiry                                               
- storage_lvm_vg_rename                                             
- storage_lvm_thinpool_rename                                       
- network_vlan                                                      
- image_create_aliases                                              
- container_stateless_copy                                          
- container_only_migration                                          
- storage_zfs_clone_copy
- unix_device_rename         
- storage_lvm_use_thinpool                                                                 
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description             
- image_force_refresh                  
- storage_lvm_lv_resizing       
- id_map_base      
- file_symlinks         
- container_push_target    
- network_vlan_physical  
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 192.168.1.112:9000
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint: 57cf720a2eea344dbaaac79e16d634b7f3ddb37b955b83a4e58dc833f543aef2
  driver: lxc
  driver_version: 2.0.9
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.10-1-dl3x0-g7-r1
  server: lxd
  server_pid: 17801
  server_version: "2.20"
  storage: btrfs
  storage_version: 4.10.2

Issue description

Running gentoo with openrc and LXD 2.20 as the host - the containers that are created as openrc containers don't get cgroups mounted.

Steps to reproduce

lxc launch gentoo-openrc test-container
lxc exec test-container mount|grep cgroup

Information to attach

Any relevant kernel output (dmesg)
Container log (lxc info NAME --show-log)

Name: test-container
Remote: unix://
Architecture: x86_64
Created: 2017/11/23 08:39 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 18855
Ips:
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
  eth0: inet6   fd42:156d:4593:a619:216:3eff:fe95:8bd0  vethKYIY8L
  eth0: inet6   fe80::216:3eff:fe95:8bd0        vethKYIY8L
Resources:
  Processes: 2
  CPU usage:
    CPU usage (in seconds): 1
  Memory usage:
    Memory (current): 14.79MB
    Memory (peak): 20.03MB
  Network usage:
    eth0:
      Bytes received: 1.21MB
      Bytes sent: 766B
      Packets received: 3570
      Packets sent: 9
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0

Log:

            lxc 20171123091239.185 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171123091239.185 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171123091239.530 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171123091239.530 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.

Container configuration (lxc config show NAME --expanded)

architecture: x86_64
config:
  image.architecture: x86_64
  image.description: Gentoo Current Generic_64 64bit  2017-11-21
  image.distribution: Gentoo
  image.name: gentoo-current-x86-64bit-generic_64
  image.os: gentoo
  raw.lxc: lxc.mount.auto = proc sys cgroup
  volatile.base_image: 049c20bfe0fcb7ab3d0ba6e59570a446662221cd2217d49bbc81488ed493e507
  volatile.eth0.hwaddr: 00:16:3e:95:8b:d0
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

The text was updated successfully, but these errors were encountered:

brauner · 2017-11-24T11:23:49Z

This has nothing to do with LXD actually. If cgroup namespaces are supported by your kernel (Which they should given your version.) liblxc will leave it up to the init system inside the container to mount cgroups or not. If openrc inside the container doesn't do it automatically or is not told to do so then there's nothing LXD can do.

palica · 2017-11-24T11:30:54Z

How can I tell openrc to mount it inside container?

brauner · 2017-11-24T11:36:13Z

I'm not an openrc expert but https://wiki.gentoo.org/wiki/OpenRC/CGroups .

palica · 2017-11-24T11:38:55Z

OK, thanks. Will look into it and report back. Is discuss a better platform for this or should I use github issues?

brauner · 2017-11-24T11:45:26Z

Discuss would probably be better for this. :) But are cgroups mounted on your gentoo with openrc host?

palica · 2017-11-24T11:50:17Z

Yes, they are.

host# mount|grep cgroup

mount|grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
rdma on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)

---SNIP---
├── openrc                                                                          
│   ├── cgroup.clone_children                                              
│   ├── cgroup.procs                                                                   
│   ├── cgroup.sane_behavior                                                     
│   ├── dhcpcd                                                                      
│   │   ├── cgroup.clone_children                                                      
│   │   ├── cgroup.procs                                                               
│   │   ├── notify_on_release                                                                   
│   │   └── tasks                                                             
│   ├── fcron                                                                 
│   │   ├── cgroup.clone_children
│   │   ├── cgroup.procs
│   │   ├── notify_on_release
│   │   └── tasks
│   ├── haveged
│   │   ├── cgroup.clone_children
│   │   ├── cgroup.procs
│   │   ├── notify_on_release
│   │   └── tasks
│   ├── lxc
│   │   ├── test-container
│   │   │   ├── cgroup.clone_children
│   │   │   ├── cgroup.procs
│   │   │   ├── notify_on_release
│   │   │   └── tasks
│   │   ├── cgroup.clone_children
│   │   ├── cgroup.procs
---SNIP---

stgraber · 2017-11-24T16:12:54Z

You said that this is an issue for openrc containers, does that mean that other containers successfully mount their own cgroups?

palica · 2017-11-24T20:42:41Z

For example xenial image from images: after mounting cgroup for systemd on the host starts without problems.

On host:

# mkdir -p /sys/fs/cgroup/systemd
# mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd

Container:

/dev/bcache48 on / type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=947,subvol=/var/lib/lxd/storage-pools/default/containers/ubu-test-1/rootfs)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,uid=100000,gid=100000)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,relatime)
udev on /dev/fuse type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
devpts on /dev/console type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=100005,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=100005,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=100000,gid=100000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755,uid=100000,gid=100000)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755,uid=100000,gid=100000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

palica · 2017-11-24T20:44:13Z

This is the mount from the host:

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=18566890,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on / type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
tmpfs on /run type tmpfs (rw,nodev,relatime,size=14854072k,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
rdma on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
/dev/bcache48 on /storage type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=5,subvol=/)
/dev/bcache48 on /usr/src type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=652,subvol=/usr/src)
/dev/bcache48 on /var type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=636,subvol=/var)
/dev/bcache48 on /srv type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=653,subvol=/srv)
/dev/bcache48 on /home type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=655,subvol=/home)
/dev/bcache48 on /root type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=656,subvol=/home/root)
/dev/bcache48 on /opt type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=654,subvol=/opt)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /var/lib/lxd/shmounts type tmpfs (rw,relatime,size=100k,mode=711)
tmpfs on /var/lib/lxd/devlxd type tmpfs (rw,relatime,size=100k,mode=755)
/dev/sda2 on /var/lib/lxd/devices/fun-bin-generic/disk.usr-local-portage type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
/dev/sda2 on /var/lib/lxd/devices/laminar-ci/disk.usr-local-bin type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
/dev/sda2 on /var/lib/lxd/devices/laminar-ci/disk.usr-local-portage type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
/dev/bcache48 on /var/lib/lxd/devices/laminar-ci/disk.var-git type btrfs (rw,noatime,compress=lzo,ssd,space_cache,subvolid=636,subvol=/var/git)
/dev/sda2 on /var/lib/lxd/devices/odoo-11/disk.usr-local-portage type f2fs (rw,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,relatime,name=systemd)

brauner · 2017-11-24T21:20:18Z

As I said, with cgroup namespaces it is up to the init system to mount cgroups not liblxc.

palica · 2017-11-24T21:36:00Z

OK, will check OpenRC. Thanks for help.

palica · 2017-11-25T14:11:16Z

Just for reference, if someone knows how to solve this on OpenRC I have opened couple of threads.

See:
LXD Forum - https://discuss.linuxcontainers.org/t/running-lxd-an-openrc-container-on-a-openrc-system-trouble-with-cgroups/843
OpenRC isue - OpenRC/openrc#187

palica · 2017-11-26T10:16:03Z

If openrc should be in charge of mounting cgroups, there is a solution to this (workaround). I have put my workaround in the issue report on OpenRC so go and read it there, please.

stgraber · 2017-11-29T20:42:34Z

Right, LXC/LXD's behavior is to setup the cgroup namespace in a way where the init system can do its normal cgroup configuration job. On systems which lack the cgroup namespce, lxcfs will then be used to fake a cgroup namespace through pre-mounted fuse.

So looks like everything is working as designed here, except for the part where openrc should actually try to mount the cgroups inside a container (as systemd does).

palica · 2017-11-29T20:53:24Z

Fixed in openrc

OpenRC/openrc@3de6395

brauner · 2018-03-12T19:34:23Z

@palicia, for future reference I've also added logic to LXC to allow users to force cgroup mounting even when cgroup namespaces are enabled.

palica · 2018-03-12T20:40:39Z

how do you do that? do you have any doc?
thank you

brauner · 2018-03-12T21:00:08Z

With LXC 3.0 you can do: lxc.mount.auto = cgroup:rw:force

palica mentioned this issue Nov 25, 2017

cgroups don't get mounted inside LXD container OpenRC/openrc#187

Closed

stgraber closed this as completed Nov 29, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LXD on OpenRC host not mounting cgroups in containers #4052

LXD on OpenRC host not mounting cgroups in containers #4052

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

stgraber commented Nov 24, 2017

palica commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

palica commented Nov 25, 2017

palica commented Nov 26, 2017

stgraber commented Nov 29, 2017

palica commented Nov 29, 2017

brauner commented Mar 12, 2018

palica commented Mar 12, 2018

brauner commented Mar 12, 2018

LXD on OpenRC host not mounting cgroups in containers #4052

LXD on OpenRC host not mounting cgroups in containers #4052

Comments

palica commented Nov 24, 2017

Required information

Issue description

Steps to reproduce

Information to attach

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

stgraber commented Nov 24, 2017

palica commented Nov 24, 2017

palica commented Nov 24, 2017

brauner commented Nov 24, 2017

palica commented Nov 24, 2017

palica commented Nov 25, 2017

palica commented Nov 26, 2017

stgraber commented Nov 29, 2017

palica commented Nov 29, 2017

brauner commented Mar 12, 2018

palica commented Mar 12, 2018

brauner commented Mar 12, 2018