Add support for gVisor within containerd

[sathieu]

What would you like to be added:

Kubespray already has support for Katacontainers within containerd. gVisor is secure and more light-weighted.

Doc: https://gvisor.dev/docs/user_guide/containerd/quick_start/

Why is this needed:

Security with performance.

[cristicalin]

@sathieu does gVisor have a stable release we could reference? From what I see in the docs they recommend latest and on the GitHub releases page there are only weekly builds. How should we go about supporting a specific version?

[sathieu]

@cristicalin There are tagged release in github: https://github.com/google/gvisor/releases

THere are also documented at https://gvisor.dev/docs/user_guide/install/#specific-release.

[cristicalin]

It's not immediately clear to me if any of those are considered stable or development releases. Should we settle on a specific date release?

[sathieu]

The doc says:

For production use, the latest release is recommended

But, I couldn't find a way to pin a specific version 🧷.

@fvoznika @ianlewis @amscanne Is it possible to pin a specific gVisor version? This is needed for Kubespray.

[ianlewis]

How are you planning to install it? You can pin versions with the debian package the normal way AFAIK. Otherwise you can use a specific version by specifying the date (however not all dates necessarily have releases)
https://gvisor.dev/docs/user_guide/install/#specific-release

[cristicalin]

@ianlewis the issue I see is that we cannot release with the version pointing at latest as that could break after release, so personally I would prefer we have a guideline on how to pick a stable version that can be used.

For example, we tag our next release 2.17 and use version latest. When we release, latest might be 20210615 but when some user deploys 2.17 in August, latest may be 20210821 and that would be a version on which we have not done any testing and they may get a broken deployment for which we cannot offer support.

@floryut or @champtar I would appreciate your thoughts here as well. I'm willing to work on the support for this but it's not clear in my mind how to properly align gVisor releases with kubespray's

Second question, I see gVisor only publishes packages for debian/ubuntu. Should we try to support RHEL derivatives or other OSes as well?

[ianlewis]

@cristicalin Right. At the link I posted it describes how you can download the binaries for a specific release at https://storage.googleapis.com/gvisor/releases/release/${yyyymmdd}/${ARCH}. Admittedly it's not yyyymmdd and rather the yyyymmdd.<version> part of the release tag name found on the releases page. Will fix those instructions shortly.

For example, the runsc binary for the May 10th release is at https://storage.googleapis.com/gvisor/releases/release/20210510.0/x86_64/runsc. The binaries at these specific urls won't change over time.

[ianlewis]

@cristicalin With regard to rpm packages, we don't have plans for RHEL derivatives yet but we can consider it if there is need. Feel free to submit an issue.

The binaries in the binary release should work however as all binaries are static binaries and should run on any system given the right OS (Linux) and architecture (x86_64).

[cristicalin]

@ianlewis I could use some help on the implementation. I got containerd to work as per gVisor docs but I cannot get docker to run as a container manager.

While docker run --runtime=runsc alpine works on any worker node, a pod launched with the runtimeClass: gvisor fails with the following error:

vents:
  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               8m56s                   default-scheduler  Successfully assigned default/nginx-gvisor to k-node-3.kaveman.intra
  Warning  FailedCreatePodSandBox  3m25s (x26 over 8m55s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "runsc" not supported
  Warning  FailedCreatePodSandBox  10s (x4 over 49s)       kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "runsc" not supported

Any help would be appreciated.

[ianlewis]

@cristicalin Unfortunately, while Docker does have the concept of "runtimes", the Docker CRI runtime doesn't support RuntimeClass. It will only work when using a CRI implementation that supports RuntimeClass, like containerd.

Another thing to worry about is that kubeadm prefers Docker as the CRI runtime if both Docker and containerd are installed so you need to make sure that the kubelet is using containerd. Some more info is written gVisor's FAQ.-.
https://gvisor.dev/docs/user_guide/faq/#runtime-handler

[cristicalin]

@ianlewis thanks for the info about the docker limitation.

I modified the change and focused on getting cri-o support instead. I referenced some information in the only issue I could find about cri-o support for gVisor (google/gvisor#2219) but I was still unable to get cri-o + gVisor combination to work. Is this combination supported by gVisor or we should limit our support to just containerd ?

[ianlewis]

@cristicalin cri-o has very limited support right now and isn't well tested. I think it would be best to limit support to containerd as we're reliant on it's shim support right now.

[cristicalin]

I restricted the use to just containerd for the moment but left in the changes to the cri-o role as well for a future when gVisor will also support it.

I'm now stuck on the the role testing via molecule.

It seems the following command gets stuck and never returns:

# ctr run --runtime io.containerd.runsc.v1 docker.io/library/hello-world:latest gvisor1

In addition, deleting the container via ctr c rm gvisor1 fails to terminate the runsc processes. Upon further digging through the gVisor issues I came across google/gvisor#4544 which describes this exact issue but does not have a fix yet.

@ianlewis can you suggest an alternative way of testing that the runtime is functional until the bug in gVisor is fixed?

[cristicalin]

Finally I managed to use crictl to implement testing though it is a bit cumbersome due to the intentionally rudimentary implementation of crictl.

The PR for this feature is in good shape for review now, I welcome your scrutiny.

[ianlewis]

You did it the right way which is to use crictl. ctr provides a --runtime option but it doesn't actually work because containerd omits sending options to the shim for esoteric reasons (shim options (plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options is technically a CRI option so it doesn't send it to the shim when using ctr because it doesn't go through the CRI plugin) so the shim is never given the ConfigPath to runsc.toml.