Docker image improvements

[juanfont]

This PR ditches distroless for Docker, and creates the /var/run/headscale path so user do not have issues when using the default config.

[ohdearaugustin]

Why would you ditch distroless?

[kradalby]

We do not use docker, and it was causing us a lot of stress and problems, so we decided that it was not worth it.

[ohdearaugustin]

But a lot of people do and I can take a look at it. This attitude of yeah we just don't use it, so we don't care is not really nice, because I also but some time into getting those container running nice. BTW: It was your commit which broke the containers. just for the deb package. This is the same argument.

[ohdearaugustin]

This PR made the container even worse then beforehand. I would rather then go with option 3 and let people configure it before you ditch containerless, which adds a lot of security to the container.

[kradalby]

I appreciate that, but it is truly a roadblock for us, and switching to "full" Debian doesnt really affect the container, while it makes it a lot better for us, it is not a breaking change.

The .deb package is something we support as it made it easier to run the binary in the way we propose, so it took priority.

[ohdearaugustin]

It is not a roadblock. A full debian container makes the images huge and now you have the problem of never knowing which debian is used inside of it. As it is against best pratice to use a "lastest style" container.

[ohdearaugustin]

This change will degrease the container security, reverts best practices for containers and will increase the size of the container without any improvements.

Edit: grammar

[juanfont]

The "we don't use" is not "we don't care". It is "we don't know how to support users having issues with that, because it is not our daily driver".

There are two things users have systematically been facing problems with, reverse proxies and Docker - way way way more than anything else, and usually unrelated to Headscale itself. We even added dedicated channels in Discord because of that.

So both Docker and reverse proxies are a bit of a sensitive topic :(

--

Yesterday when I was looking at the headscale.sock issue I could not troubleshoot the problem because the container did not provide a shell. In the end I tried to think of reasons for keeping distroless. And there were two points:

• Storage savings are super minor (20MB). I don't think it is really worth the pain for users and for us if it is because of storage savings.

• Security: if the threat model of one of Headscale users is an RCE in a Go application + container escape... that's state-sponsored grade. I don't know, I feel I am doing something financially very wrong by doing this for free 😅

Then I had a look at all the usual apps I run at work, in a pretty standard IT environment (some docker, some compose, and Kubernetes) and counted how many use distroless: zero. So we would not be such an outlier...

--

So it is just "we don't use it".

[ohdearaugustin]

Yeah the main problem is that a huge portion of the people, who use docker. Unfortunately do not understand docker in the first place, even huge companies. The reverse proxy is not really related to the docker as far as I know.

This change won't fix either of those problems.

• Now the program is running as root user

But yeah in the end it is your project. I just would love to keep the distroless one in some way.

BTW: The debug image is exactly for this kind of thing when you need a shell in the container.

[juanfont]

I mentioned the reverse proxy because it's also a (the other?) source of user issues. That's indeed the only relation to Docker regarding Headscale.

--

Before the change I also did a bit of search on the interwebs - as I am no Docker expert either. And I could not see a 100% definitive answer on distroless. We are far from being a total outlier here, with reasonable-sounding people challenging the supposed advantages.

That being said... we can make a decision, but I am more than happy to be corrected and switch ways.

The thing is that from the distro packaging people we were told we should use /var/run/headscale, not /var/run/.

And on the other hand, I could not even run mkdir in the Dockerfile to align there.

--

The debug image comes with Go tools, it is an order of magnitude larger than distroless or the new one.

Distroless: 19MB
Debian 11-based: 39MB
Debug: 300MB

[gabe565]

I understand that you guys are at a point where you want to get rid of distroless, and I get it if it causes a maintenance burden, but I want to add that the distroless images are best practice with respect to security. You're correct that they're used less commonly, but it feels to me that they've been gaining traction recently. It also feels more important for a VPN server to maintain a very small attack vector.

My Headscale Helm chart gets regular security scans since it's listed in ArtifactHub, and while these scan results are always a bit overzealous, this change added a lot of new vulnerabilities including 1 critical. I want to be clear that this doesn't mean Headscale is vulnerable to anything, just that there are more files in the container that could possibly be taken advantage of.

• v0.22.1 scan results
• v0.22.2 scan results

I also want to mention that it would be possible to make the debug image much smaller! Instead of the debug image using golang:1.20.0-bullseye, you could use gcr.io/distroless/base-debian11:debug which includes standard cli tooling, and is only ~1MB larger than the non-debug image:

❯ docker image ls | grep distroless
gcr.io/distroless/base-debian11   debug-nonroot   ed4d800c72f7   N/A             18.7MB
gcr.io/distroless/base-debian11   nonroot         96896139ac21   N/A             17.5MB

[gabe565]

What about making gcr.io/distroless/base-debian11:debug be the default, with another one called something like minimal which uses the smaller image?

Or even using alpine would most likely be smaller and more secure than debian. I wouldn't mind making a PR to update the Dockerfile if you guys want to see what that would look like. Just let me know what you think!

[juanfont]

I am afraid this is the way it is going to be for now, until we have the time to ensure we can change it and it will keep the quality and quality of life we require.

[alexhalbi]

Hi,
I also had some thoughts about what can be done to have a flexible way to use the distroless container and be able to easily and interactively configure headscale inside of the container. (especially in different environments like Azure Container Instances, which does not support executing with arguments inside of containers.)

I my opinion it is not needed to have a full blown shell inside of the container, since the config file should be mapped inside the container anyways.

Therefore I would propose to have a second executable inside of a distroless container (could also be called minimal as proposed by @gabe565) which provides an interactive shell to headscale itself which is just executing the input using the headscale executable in the container.

> create user test

Will be executed as:

headscale create user test

And the output will be redirected to the user followed by another prompt.

Using that method headscale can be easily configured inside the container, without introducing unneccesary functions to the container, like a full blown shell and all the other unneccesary things outside distroless.

What do you think about that? I could also write the code needed for that and create a PR.