Unable to run fluent-bit as non root

[tombart]

Run fluent-bit as non root
I'm trying to increase security of my k8s cluster and would like to run all monitoring and logging tools as non root. Currently fluent-bit is running as root by default. Is it possible to update the base image with fluent bit user and enable fluent bit to read logs from k8s/docker as that user?
The problem:
Docker logs are stored in /var/lib/docker directory (when you use default json logger plugin) which has root read permissions only. When I've created fluent bit user it couldn't read any .log files from that directory. Even if I managed to change permission of current container .log files as soon as you deploy new container on that host the logs for that container are written as root. Does anyone have any solution? Maybe direct docker container logs to other directory (couldn't find any information on that)?

Additional context
I'm running k8s cluster, with fluent-bit running in docker.Fluent-bit input plugin configured to point to /var/logs (which is simlinked to /var/lib/docker - internal directory of docker where docker deamon and container logs are stored)

[edsiper]

hi,

I think the issue is not running Fluent Bit as non-root, there is a permission/access problem. Fluent Bit can work as normal user without problem but you have to make sure that it have access to the files from the mounted volume.

I don't know how to configure docker to create readable logs for normal users.

[sshishov]

Nice, fixed :) But where is documentation or at least link for the solution?

[harrisjoseph]

I've added the following to my daemonset container, to remove all unused capabilities when running as root:

        securityContext:
          capabilities:
            drop:
              - ALL
            add:
              - CAP_FOWNER

The container only needs file-owner to read files on the host, all other root capabilities are dropped.

It's not quite running as non-root, but using a minimally powerful root user is better than the default.

[amitkatyal]

@harrisjoseph , I tried running as a non-root user and CAP_FOWNER capability but it doesn't help. the fluent-bit tail plugin is still not able to read the files. Any other suggestions?

[harrisjoseph]

The container still needs to run as root to use the FOWNER capability. Setting the securityContext & capabilities is a way of removing any unused capabilities in the cases where a container needs to run as root. It's a bit of a compromise, and I'd rather be able to run as an actual non-root user, but this approach closes a lot of the security holes that come with running as unrestricted root.

[amitkatyal]

If I run the container as root and drop all the capabilities, then it works fine even w/o adding the CAP_FOWNER capability.

[harrisjoseph]

That's very interesting. Does running getpcaps {pid} on the host show that the process has no capabilities?

[amitkatyal]

Please find below the caps. Since the container is running as root, will it not have the privileges to read the files owned by the root?