podman support

[mmguero]

I don't think there's any reason Malcolm couldn't run fine with rootless podman, particularly using the --userns keep-id:uid=xxxx,gid=yyyy option to map the current host user to a specific ID inside the container.

Just need to work through it and resolve issues that crop up.

[mmguero]

(copy/paste from a teams conversation, apologize for the bad formatting)

one bit of info I should mention about the podman one: all of the malcolm docker images go through the same entrypoint, this script. the purpose of this script is, for the most part, to set up the uid/gid stuff inside the container to match what the user has specified in the config environment variables. Some containers have this environment variable PUSER_PRIV_DROP set to true (like this) to immediately drop privileges in that script, while others need to be root for just a bit longer and then drop privileges inside the container either inside a supervisord process or some other way in the startup code of that container. But the important part is that every single one of the containers actual runtime processes drops to the correct non-root user upon startup. This ensures, among other things, that all files created get created from the host (the machine running Docker) point of view with the correct file ownership: if user manny is running Docker and that UID is 1000, then all processes run under and all files are created with ownership 1000; if user seth is running it and his UID is 1001, all processes run under 1001 and ownership of files belongs to that UID.

so the way docker and podman work is just a bit different: for docker, the PUID and PGID variables should be set to whatever your users' are, e.g., 1000 or whatever, and they'll get set appropriately inside the container. however, for rootless podman, it works a little bit different: by default, uid/gid 0 and 0 (or, in other words, root) map to the UID/GID of the user running podman.

So for like 90% of the containers, this can just be solved by setting the variables in process.env to 0 and 0 and it will pretty much work.

however, there are some containers that don't like this: for example, the opensearch container refuses to start if the UID/GID is set to zero, saying it's not safe to run as root.

however, podman has a userns mode capability with keep-id (here are some examples) that I think will take care of the problem. in other words, I think the solution is to set process.env to be the same UID and GID as we would with docker, but use the keep-id mode to map it so 1000 inside the container is the same as 1000 outside the container (or whatever the UID is, it would need to work for whatever is set in that .env file). here is some other reading on userns modes.

so one of the tricks will be how to set the --userns keep-id (or whatever the exact flag looks like) into the environment running podman via podman-compose or docker compose (i think both podman-compose and docker compose should be supported if possible, as both are valid options for running podman against docker-compose files). here are some things to look at:

• does podman-compose have a way to pass the --userns keep-id argument
• read PODMAN_USERNS env variable when cerating containers through compose API
• "since version 3.7 the docker compose file specification allows defining extension..."
• podman-compose tests/uidmaps example

Basically, it needs to work correctly for whatever the user's UID and GID from process.env are, and it needs to be able to do so ideally without modifying the docker-compose.yml file directly (ie., just pulling it from process.env) and it needs to not break existing docker-supported functionality

[mmguero]

Podman 5 is out which seems to have a bunch of improvements, I'd recommend we use that as a baseline (and, if there's anything it makes easier, making that a minimum requirement).

[mmguero]

I've started playing with this, and honestly the issues are very few:

• Running podman rootless, without any of the UID mapping stuff from the previous comment, UID 0 maps to the host user's UID. So for the most part, setting UID and GID to 0 during configuration "just works."
• the local driver for logging is fine for Docker but not for podman. Locally I changed it to journald but we need to determine what the right thing to do is, and how to handle it (I'd rather not have to maintain two different compose .yml files, although we may have to).
• The biggest concern seems to be opensearch, because it simply will not let you run as root (I've even tried taking that out of the entrypoint, but there are other safeguards on it elsewhere. This means that even though it's not really running as root, inside of podman, it thinks it is. I think we're going to have to do some of the userns mapping mentioned above for this one.
  • In the meantime, I've pointed it at an external elasticsearch instance and things come up correctly

[mmguero]

Here's a thread that's maybe working for the user namespace mapping stuff:

• I've added a way to specify the container runtime (e.g., podman) via a -r/--runtime option or a MALCOLM_CONTAINER_RUNTIME environment variable (see mmguero-dev/Malcolm@d64970a).
• This means that the scripts will execute podman compose instead of docker compose, and, as the default warning message says when executing podman compose says, "Executing external compose provider "/usr/local/bin/docker-compose". This is not a bad thing, because...
• Adding userns_mode: keep-id to the services in the compose .yml file seems to now do the mapping correctly.

Need to 100% confirm this, but it looks good. Also need to make sure that this is ignored when in regular docker-compose mode. If not, we can add it on the fly, but that's less fun.

[mmguero]

See containers/podman#23347 for another issue. Trying to run via podman compose with userns_mode: keep-id. Only the VOLUME-declared paths have 999 ownership.

[mmguero]

I've got this in a pretty-much-working state, just going through and hand-checking the logs and whatnot to make sure everything is the way it should be. I still need to test live-capture as well.

[mmguero]

As expected live capture is failing, but everything else is working correctly as far as I can tell.

[mmguero]

I have determined that it's impossible to do the live network capture using rootless podman (probably rootless docker as well) because of the user namespacing. Even with tricks like setcap and --cap-add, etc., it still results in Operation Not Permitted.

I've added this to the documentation in a few of the places we mention podman:

It should be noted that if rootless Podman is used, Malcolm itself cannot perform traffic capture on local network interfaces, although it can accept network traffic metadata forwarded from a a network sensor appliance.