Shiftfs on AWS AMI ... #121

ctalledo · 2020-11-15T20:27:50Z

ctalledo
Nov 15, 2020
Maintainer

I'm having a hard time finding an AMI that has shiftfs so that I can use the sysbox runtime in EC2.

Yes, unfortunately the AMIs for Ubuntu don't carry shiftfs (yet). There is a out-of-tree distribution of shiftfs that you can install and load into the kernel, though I've not personally tried it but I think @rodnymolina may have. See here. The kernel must still be Ubuntu and >= 5.0 (e.g., ubuntu focal).

@rodnymolina : did you ever get a chance to try shiftfs dkms on an AWS Ubuntu-based AMI?

ScottG489 · 2020-11-15T20:52:02Z

ScottG489
Nov 15, 2020

@ctalledo Thanks for setting up this thread.

Just wanted to mention first that while I am currently using an Ubuntu AMI, I'd be open to try another AMI within reason.

In the mean time, I'll take a look at toby63/shiftfs-dkms. Appreciate the help.

13 replies

ScottG489 Nov 16, 2020

Here's a reproduction. Note that in both of these cases I've already followed the installation instructions and installed the sysbox_0.2.1-0.ubuntu-focal_amd64.deb package on the host with userns-remap of course needing to be enabled since I have no shiftfs support.

Without --userns=host:

$ sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock ubuntu bash
root@492c9b3323b8:/# apt-get update && apt-get install -y docker.io
[update/install lines omitted]..........
root@492c9b3323b8:/# docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

With --userns=host:

$ sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock --userns=host ubuntu bash
root@47e05c3aa491:/# apt-get update && apt-get install -y docker.io
[update/install lines omitted]..........
root@db6d301deedc:/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
db6d301deedc        ubuntu              "bash"              About a minute ago   Up About a minute                       festive_solomon
root@db6d301deedc:/# docker run -it --runtime=sysbox-runc ubuntu bash
root@9fa1acb7423f:/# apt-get update && apt-get install -y docker.io
[update/install lines omitted]..........
root@9fa1acb7423f:/# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
root@9fa1acb7423f:/# dockerd > /var/log/dockerd.log 2>&1 &
[1] 4100
root@9fa1acb7423f:/# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@9fa1acb7423f:/# # For extra effect...
root@9fa1acb7423f:/# docker run -it --runtime=sysbox-runc ubuntu bash
docker: Error response from daemon: Unknown runtime specified sysbox-runc.
See 'docker run --help'.
root@9fa1acb7423f:/# docker run -it -v /var/run/docker.sock:/var/run/docker.sock ubuntu bash
root@ebeffd498379:/# apt-get update && apt-get install -y docker.io
[update/install lines omitted]..........
root@ebeffd498379:/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
ebeffd498379        ubuntu              "bash"              About a minute ago   Up About a minute                       blissful_varahamihira

So without --userns=host on the outer-most container I can't really get anywhere. With --userns=host I can see the hosts containers as normal. From there it's what I imagine would be standard usage of the sysbox runtime. I start a container using the runtime, start the docker daemon, and you can see that the container is as expectedly sandboxed as it only sees it's own containers. For a little extra I show that although you can't nest another sysbox container (as expected) you can nest containers using docker.sock and they only see what's at the root of their sysbox container.

All of this works the same without --userns=host on a host with shiftfs support. Sorry if that was a bit verbose. Lots of nesting...

Now my only question is: what are the ramifications of using --userns=host on the outer-most container?

ScottG489 Nov 16, 2020

Hey @ctalledo. Only saw your comment after I hit "reply".

So it sounds like the only effect of using --userns=host is that it's using the userns (host) that it would normally use before going through with the sysbox install?

From reading the docker docs to Disable namespace remapping for a container it sounds like there's more to it:

There is a side effect when using this flag: user remapping will not be enabled for that container but, because the read-only (image) layers are shared between containers, ownership of the containers filesystem will still be remapped.

What this means is that the whole container filesystem will belong to the user specified in the --userns-remap daemon config (231072 in the example above). This can lead to unexpected behavior of programs inside the container. For instance sudo (which checks that its binaries belong to user 0) or binaries with a setuid flag.

Thoughts?

ctalledo Nov 16, 2020
Maintainer Author

For your particular use-case, I think --userns=host appears to be fine given that the CI server runs without problem. Also, using this flag does not pose any security risk beyond what a regular docker container would (and since you are already giving your CI container access to the host's Docker socket, it's assumed that you trust the CI container).

Now, the text you quoted from the docker docs is referring to the following:

When Docker is configured with userns-remap, the containers use the Linux user-namespace. This namespace works by mapping user/group-IDs between container and the host. For example, the root user in the container may map to user-ID 165536 on the host. In order for this to work, Docker sets the ownership of the container's root filesystem to user 165536:165536 at host level (such that from within the container the files show up as root:root).

Now, when you use --userns=host, Docker no longer uses the Linux user-namespace for that container. As a result no user-ID mapping occurs (i.e., root user in the container ** is ** the root user in the host). But (unfortunately) the container's root filesystem continues to have the ownership of the non-root user 165536:165536. Thus, from inside the container files show up with 165536:165536 instead of root:root as would be expected:

/ # ctalledo@nestybox-srv-01:~$ docker run --userns=host -it --rm alpine
/ # ls -l /
total 56
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 bin
drwxr-xr-x    5 root     root           360 Nov 16 01:45 dev
drwxr-xr-x    1 165536   165536        4096 Nov 16 01:45 etc
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 home
drwxr-xr-x    7 165536   165536        4096 Oct 21 09:23 lib
drwxr-xr-x    5 165536   165536        4096 Oct 21 09:23 media
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 mnt
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 opt
dr-xr-xr-x  978 root     root             0 Nov 16 01:45 proc
drwx------    1 165536   165536        4096 Nov 16 01:45 root
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 run
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 sbin
drwxr-xr-x    2 165536   165536        4096 Oct 21 09:23 srv
dr-xr-xr-x   13 root     root             0 Nov 16 01:45 sys
drwxrwxrwt    2 165536   165536        4096 Oct 21 09:23 tmp
drwxr-xr-x    7 165536   165536        4096 Oct 21 09:23 usr
drwxr-xr-x   12 165536   165536        4096 Oct 21 09:23 var

This is not ideal, and can cause problems for some programs as the Docker docs mentioned. It appears that in your case it's not causing problems, which is good.

Note that from within the container you can always do a chown -R root:root bin dev etc home lib media mnt opt root run sbin srv tmp usr var and things go back to normal:

/ # chown -R root:root bin dev etc home lib media mnt opt root run sbin srv tmp usr var
/ # ls -l /
total 84
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 bin
drwxr-xr-x    5 root     root           360 Nov 16 02:05 dev
drwxr-xr-x    1 root     root          4096 Nov 16 02:05 etc
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 home
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 lib
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 media
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 mnt
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 opt
dr-xr-xr-x  971 root     root             0 Nov 16 02:05 proc
drwx------    1 root     root          4096 Nov 16 02:05 root
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 run
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 sbin
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 srv
dr-xr-xr-x   13 root     root             0 Nov 16 02:05 sys
drwxrwxrwt    1 root     root          4096 Oct 21 09:23 tmp
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 usr
drwxr-xr-x    1 root     root          4096 Oct 21 09:23 var

It's a bit of a hack but it works :)

ScottG489 Nov 16, 2020

Wow very nice explanation! Appreciate all the help from you and @rodnymolina.

Once I get this feature settled in my project I'll give the shiftfs-dkms setup a shot. Let me know if you ever end up getting that built into the installer. I'd be happy to test out something you have in beta.

In case you're curious here's my project:

ScottG489/conjob

It's a lightweight REST service for containerized jobs which can be run in a serverless fashion. Sysbox is a real game changer for me because my largest bug has been finding a way to keep the containerized jobs completely segregated. Thanks so much and good luck with your project!

ctalledo Nov 16, 2020
Maintainer Author

Very cool, thanks for sharing! Glad Sysbox was able to help.

Let us know of any more feedback (feature requests, more use-cases, bug reports, etc.) as it really helps us understand how people are using Sysbox and thus how to improve it.

rodnymolina · 2020-11-15T22:16:28Z

rodnymolina
Nov 15, 2020
Maintainer

@ScottG489, i just gave it a try on an Ubuntu Focal EC2 VM. Find below the required steps to build your own 'shiftfs' module -- pretty much the same ones mentioned on shiftfs-dkms repo.

One question for you. Would you rather do this process yourself (so that you fully know what's underneath)? or would you prefer to have Sysbox installer take care of all this for you (even though installation process will now take some extra time to fetch dkms and build shiftfs)? What's your take on this? Thanks.

# Verify that 'shiftfs' module is not already present.
$ lsmod | grep shiftfs

# We'll need dkms package to build shiftfs.
$ sudo apt-get update && sudo apt-get install dkms -y

$ git clone [email protected]:toby63/shiftfs-dkms.git

$ cd shiftfs-dkms/

$ sudo ./update1

$ sudo make -f Makefile.dkms

# Verify that 'shiftfs' module has been properly installed in modules path:
$ sudo find /lib/modules/$(uname -r)/ -iname "*shiftf*"
/lib/modules/5.4.0-1021-aws/updates/dkms/shiftfs.ko

# Load module and check that is properly loaded:
$ sudo modprobe shiftfs
$ lsmod | grep shiftfs
shiftfs                28672  0

1 reply

ScottG489 Nov 15, 2020

Thanks, this is great! Will give it a try. See my recent comment above on a workaround I think I found in the meantime.

I think ideally it would be nice for the installer to take care of it for you. I think if I was coming across this project anew and the installer took a while it wouldn't bother me too much. I think it working out of the box in most situations is the best approach. However, it might be the better idea to have it prompt the user in the same way you prompt them to install with userns-remap.

That said, I'm not sure how hacky of a workaround this is or what other implications or side effects there could be. It also of course should be well documented.

viatcheslavmogilevsky · 2021-02-11T11:07:33Z

viatcheslavmogilevsky
Feb 11, 2021

I have packer config to make Ubuntu AMI + shiftfs - works smoothly (kudos to @ctalledo for inspiration of how to configure json files via jq)

Here are some notes on installing shiftfs

the sudo ./update1 complains:

Don't run this as root!

Also I had add permission to execute:

chmod +x ./update1
sudo ./update1

and configure module to launch on instance startup (assuming we have systemd):

echo "shiftfs" | sudo tee /etc/modules-load.d/shiftfs.conf

0 replies

viatcheslavmogilevsky · 2021-02-12T19:55:54Z

viatcheslavmogilevsky
Feb 12, 2021

here is example of packer config, to run this packer hcl you have to install ssm session manager https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html and packer

variable "ami_name" {
  type = string
}

variable "region" {
  type    = string
}

source "amazon-ebs" "this" {
  ami_name      = "${var.ami_name}"
  instance_type = "t2.medium"
  region        = "${var.region}"
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["099720109477"]
  }
  launch_block_device_mappings {
    device_name = "/dev/sda1"
    volume_type = "gp2"
    volume_size = 50
    delete_on_termination = true
  }
  ssh_username = "ubuntu"
  ssh_interface = "session_manager"
  communicator = "ssh"
  temporary_iam_instance_profile_policy_document {
    Version = "2012-10-17"
    Statement {
      Effect = "Allow"
      Resource = ["*"]
      Action = [
        "ssm:DescribeAssociation",
        "ssm:GetDeployablePatchSnapshotForInstance",
        "ssm:GetDocument",
        "ssm:DescribeDocument",
        "ssm:GetManifest",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:ListAssociations",
        "ssm:ListInstanceAssociations",
        "ssm:PutInventory",
        "ssm:PutComplianceItems",
        "ssm:PutConfigurePackageResult",
        "ssm:UpdateAssociationStatus",
        "ssm:UpdateInstanceAssociationStatus",
        "ssm:UpdateInstanceInformation",
      ]
    }
    Statement {
      Effect = "Allow"
      Resource = ["*"]
      Action = [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel",
      ]
    }
    Statement {
      Effect = "Allow"
      Resource = ["*"]
      Action = [
        "ec2messages:AcknowledgeMessage",
        "ec2messages:DeleteMessage",
        "ec2messages:FailMessage",
        "ec2messages:GetEndpoint",
        "ec2messages:GetMessages",
        "ec2messages:SendReply",
      ]
    }
  }

  vpc_filter {
    filters = {
      "tag:Name": "some-private-vpc"
    }
  }
  subnet_filter {
    filters = {
      "tag:Name": "some-private-subnet"
    }
    most_free = true
  }
  temporary_security_group_source_cidrs = ["10.0.0.0/8"]
}

# a build block invokes sources and runs provisioning steps on them.
build {
  sources = ["source.amazon-ebs.this"]

# install prerequisites
  provisioner "shell" {
    execute_command = "sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    inline = [
      "apt-get update -y",
      "apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common dkms git",
    ]
  }

# shifts installation
  provisioner "shell" {
    inline = [
      "cd ~",
      "git clone git://github.com/toby63/shiftfs-dkms.git shiftfs-dkms",
      "cd shiftfs-dkms",
      "git checkout e4ddbb55b0cf804899b16058124ab39ec4747db0",
      "chmod +x ./update1",
      "sudo ./update1",
      "sudo make -f Makefile.dkms",
      "sudo find /lib/modules/$(uname -r)/ -iname \"*shiftf*\"",
      "sudo modprobe shiftfs",
      "lsmod | grep shiftfs",
      "echo \"shiftfs\" | sudo tee /etc/modules-load.d/shiftfs.conf"
    ]
  }

# docker & sysbox installation
  provisioner "shell" {
    execute_command = "sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    inline = [
      "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -",
      "apt-key fingerprint 0EBFCD88",
      "add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"",
      "apt-get install -y docker-ce docker-ce-cli containerd.io",
      "cd /tmp",
      "wget https://github.com/nestybox/sysbox/releases/download/v0.2.1/sysbox_0.2.1-0.ubuntu-focal_amd64.deb",
      "sha256sum sysbox_0.2.1-0.ubuntu-focal_amd64.deb",
      "DEBIAN_FRONTEND=noninteractive apt-get install ./sysbox_0.2.1-0.ubuntu-focal_amd64.deb -y",
      "rm -f ./sysbox_0.2.1-0.ubuntu-focal_amd64.deb",
      "groupadd -f docker",
      "usermod -aG docker ubuntu"
    ]
  }
}

to run such packer config, save content above under file name matches .pkr.hcl (for example example.pkr.hcl)
and launch command:

packer build -var region=us-east-1 -var ami_name=ubuntu-sysbox-$(date +%s) example.pkr.hcl

1 reply

rodnymolina Feb 12, 2021
Maintainer

@viatcheslavmogilevsky, excellent work, thanks for sharing this!

viatcheslavmogilevsky · 2021-02-16T13:32:06Z

viatcheslavmogilevsky
Feb 16, 2021

Just created gist for convenience: https://gist.github.com/viatcheslavmogilevsky/e5d82e7cc755f6d93a79a6cedf489719

1 reply

ctalledo Feb 16, 2021
Maintainer Author

Awesome, thanks @viatcheslavmogilevsky !

bjorntheart · 2021-09-08T15:18:59Z

bjorntheart
Sep 8, 2021

@viatcheslavmogilevsky the command git checkout e4ddbb55b0cf804899b16058124ab39ec4747db0 doesn't not work. Could be that it was in a branch that was perhaps deleted.

9 replies

bjorntheart Sep 9, 2021

@viatcheslavmogilevsky I agree 100% with @ctalledo. Thank you for sharing. @ctalledo I tried with so many kernel versions yesterday, so I'm not sure which one. I'll do some more testing and report back here

bjorntheart Sep 9, 2021

@viatcheslavmogilevsky @ctalledo The new gist worked great! I spun up an EC2 instance using ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20210415 with Kernel 5.4

ctalledo Sep 9, 2021
Maintainer Author

Hi @bjorntheart , great to hear ... glad that it worked! If you can share the updated gist here (or the changes you did), it would be great too.

And thanks again to @viatcheslavmogilevsky for sharing it in the first place.

bjorntheart Sep 10, 2021

@ctalledo I used @viatcheslavmogilevsky updated Gist here https://gist.github.com/viatcheslavmogilevsky/e5d82e7cc755f6d93a79a6cedf489719

ctalledo Sep 10, 2021
Maintainer Author

Ok got it ... thanks @bjorntheart ...

ctalledo · 2022-04-23T21:37:05Z

ctalledo
Apr 23, 2022
Maintainer Author

FYI, here is a sample Packer script for building an AMI with pre-installed Sysbox based on an Ubuntu EKS AMI, courtesy of @maximsmol.

https://github.com/latchbio/sysbox-eks-ami

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shiftfs on AWS AMI ... #121

{{title}}

Replies: 7 comments 25 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Shiftfs on AWS AMI ... #121

ctalledo Nov 15, 2020 Maintainer

Replies: 7 comments · 25 replies

ctalledo Nov 16, 2020 Maintainer Author

ctalledo Nov 16, 2020 Maintainer Author

rodnymolina Nov 15, 2020 Maintainer

rodnymolina Feb 12, 2021 Maintainer

ctalledo Feb 16, 2021 Maintainer Author

ctalledo Sep 9, 2021 Maintainer Author

ctalledo Sep 10, 2021 Maintainer Author

ctalledo Apr 23, 2022 Maintainer Author

ctalledo
Nov 15, 2020
Maintainer

Replies: 7 comments 25 replies

ctalledo Nov 16, 2020
Maintainer Author

ctalledo Nov 16, 2020
Maintainer Author

rodnymolina
Nov 15, 2020
Maintainer

rodnymolina Feb 12, 2021
Maintainer

ctalledo Feb 16, 2021
Maintainer Author

ctalledo Sep 9, 2021
Maintainer Author

ctalledo Sep 10, 2021
Maintainer Author

ctalledo
Apr 23, 2022
Maintainer Author