Rootless fluent-bit with docker input plugin

[bmodotdev]

Is your feature request related to a problem? Please describe.
I've included this as a feature request because I am not sure if running fluent-bit rootless is considered supported. If it is considered supported, then I would consider this a bug in the fluent-bit docker input plugin.

This is semi-related to issue #872, but in that issue, the reporter was running fluent-bit in a docker container and references a json input plugin. However, this issue concerns running fluent-bit on the host, and using the fluent-bit docker input plugin to gather logs from docker containers.

Setup
Host OS: Ubuntu 20.04.4 LTS
Running docker directly on the host, not in docker container.
Installed via https://docs.fluentbit.io/manual/installation/linux/ubuntu.
Customized to run rootless via systemd override.conf (ansible jinja2 template below):

[Service]
User={{ fluent_bit_user }}
PrivateTmp=true

The fluent-bit user is also added to the docker group on the host.
The fluent-bit conf includes a docker input conf as documented here: https://docs.fluentbit.io/manual/pipeline/inputs/docker-metrics.

Errors

Jun 26 20:42:23 hostname fluent-bit[2427684]: [2022/06/26 20:42:23] [error] [plugins/in_docker/docker.c:277 errno=13] Permission denied
Jun 26 20:42:23 hostname fluent-bit[2427684]: [2022/06/26 20:42:23] [error] [input:docker:docker.0] cannot open /var/lib/docker/containers/9e92d2f20c7fe85314094c72d369a14f538be17fb42b0a96be3d5637ab1cf87e/config.v2.json

Permissions of the file mentioned above:

# stat -c '%u %g %a' /var/lib/docker/containers/9e92d2f20c7fe85314094c72d369a14f538be17fb42b0a96be3d5637ab1cf87e/config.v2.json
0 0 600

Describe the solution you'd like
The fluent-bit docker input plugin should not access /var/lib/docker directly, rather it should gather logs in the same way that docker logs does:

# getent group docker
docker:x:998:fluent-bit

# su fluent-bit -c 'ls /var/lib/docker/'
ls: cannot open directory '/var/lib/docker/': Permission denied

# su fluent-bit -c 'docker logs webserver -n1'
2022/06/26 20:45:04 ...

Docker developers seem to have a strong opinion that applications should not be accessing /var/lib/docker/ directly, and if they do they do at their own risk. Even more so, it seems they have a strong opinion that if they are accessing /var/lib/docker/, the application should be doing so as root. Here is an excerpt from docker docs: https://docs.docker.com/config/containers/logging/local/

The local logging driver uses file-based storage. The file-format and storage mechanism are designed to be exclusively accessed by the Docker daemon, and should not be used by external tools as the implementation may change in future releases.

Describe alternatives you've considered

1. Run docker as rootless, i.e. username docker, and hope /var/lib/docker/ files have group docker?
2. Configure docker to use journald logging driver and switch fluent-bit docker input login for fluent-bit systemd input plugin. This kinda defeats the pupose of having a fluent-bit docker input plugin.
3. Configure docker to use fluentd logging driver and accept I cannot use fluent-bit in the scenario? Or perhaps docker's fluentd logging driver is compatible with fluent-bit?

Additional context
This affects my want to provide security in depth, by running fluent-bit rootless. As a result, I cannot use the officially supported fluent-bit docker input plugin.

I can provide ansible playbook/roles if really needed to replicate.

[bmodotdev]

I think alternative 2 works fine for me; configure docker journald log driver and switch to fluent-bit systemd input plugin. I just needed to add the custom fluent-bit user to the systemd-journal group to allow journal access, then in my fluent-bit systemd input plugin config, I added Systemd_Filter CONTAINER_NAME=webserver to cut down on noise.

This is probably better than giving the fluent-bit user access to the docker group anyway. I also had other custom systemd services I wanted to watch with fluent-bit, so it makes even more sense.

I still think approach in the fluent-bit docker input plugin is more fragile and limited since it goes against the docker developer suggestions to not reach into /var/lib/docker, but maybe there is a very good reason it's done this way.

PS: I should have begun by saying how thankful I am that ya'll provide this awesome software 😄 Thank you!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.