Podman no-priv in K8s (on Talos on Hardware)

[hh]

Exploring Rootless Podman w/o the privileged flag: https://www.redhat.com/sysadmin/podman-inside-kubernetes

Which will require we use a fuse-device-plugin similar to this one:

https://github.com/kuberenetes-learning-group/fuse-device-plugin/tree/master (is two years old...)

The fus-device-plugin deployed, and then I tried running it without a shared volume mount, but seems to need host path, which needs to be limited via some type of policy.

podman run ubi8 echo hello
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
cannot clone: Operation not permitted
Error: cannot re-exec process

This is basically the following:

rootless-no-priv.yaml

apiVersion: v1
kind: Pod
metadata:
 name: no-priv
spec:
 containers:
   - name: no-priv
     image: quay.io/podman/stable
     args:
       - sleep
       - "1000000"
     securityContext:
       runAsUser: 1000
     resources:
       limits:
         github.com/fuse: 1
##### above here only, Podman complains about missing mount with rootless containers
##### Adding below here is going to require some changes to the security policy
     volumeMounts:
       - mountPath: /home/podman/.local/share/containers
         name: podman-local
 volumes:
   - name: podman-local
     hostPath:
       path: /home/umohnani/.local/share/containers

Here is an old way with PodSecurityPolicy: https://dev.to/mxglt/kubernetes-give-rights-for-hostpath-volumes-to-services-121f
However PSP seems to have been deprecated / removed by 1.25: https://kubernetes.io/docs/concepts/security/pod-security-policy/

There is a migration path: https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/
And it seems allowedHostPaths will need to be covered by an admission web hook: https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/#eliminate-non-standard-options

May need to loop back around.

[hh]

Getting a bit further, now it's permissions... but still not a clear path.
Was hoping fsGroup and fsGroupChangePolicy would help:

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods

apiVersion: v1
kind: Pod
metadata:
 name: d
spec:
 securityContext:
   runAsUser: 1000
   runAsGroup: 1000
   fsGroup: 1000
   # fsGroupChangePolicy: "OnRootMismatch"
   fsGroupChangePolicy: "Always"
 containers:
   - name: no-priv
     image: quay.io/podman/stable
     args:
       - sleep
       - "1000000"
     securityContext:
       runAsUser: 1000
       runAsGroup: 1000
     resources:
       limits:
         github.com/fuse: 1
     volumeMounts:
       - mountPath: /home/podman/.local/share/containers
         name: podman-local
 volumes:
   - name: podman-local
     hostPath:
       path: /var/local/podman

podman run ubi8 echo hello
Error: creating runtime static files directory "/home/podman/.local/share/containers/storage/libpod": mkdir /home/podman/.local/share/containers/storage: permission denied