CRI-O: Default podman CNI config prevents use of custom network plugins

[joshmue]

Steps to reproduce the issue:

1. Download minikube 1.11.0; Install Virtualbox.
2. Run minikube start --network-plugin=cni --container-runtime=cri-o without --enable-default-cni
3. Install a custom network plugin in order to make use of Kubernetes NetworkPolicies. E. g. following these instructions for cilium (except regarding the invocation of minikube start).
4. Test NetworkPolicies according to the general documentation.
5. Notice that applied NetworkPolicies are not enforced.

Apparent cause of the issue:

It turns out that there is a default podman bridge network CNI configuration at /etc/cni/net.d/87-podman-bridge.conflist in the minikube VM which prevents the custom network plugin to be used.
If using docker as container runtime, this default configuration does not seem to cause any problems.

Workaround:

Deleting this CNI configuration, restarting CRI-O and recreating the concerned pods causes the NetworkPolicies to be enforced as expected.

minikube ssh -- sudo rm /etc/cni/net.d/87-podman-bridge.conflist
minikube ssh -- sudo systemctl restart crio
kubectl delete pods --all

The next time the minikube VM is started, /etc/cni/net.d/87-podman-bridge.conflist will be present again.

Thoughts:

Actually, am not sure if that is an issue with minikube or if cri-o just handles prioritization differently than docker. Either way, when using minikube, this behavior is unexpected, IMHO.

[priyawadhwa]

Hey @joshmue thanks for opening this issue. It seems like there's a workaround, but that the behavior isn't intuitive. Maybe adding to documentation is the solution here.

I'm not super familiar with CNI, maybe @medyagh or @tstromberg could weigh in?

[afbjorklund]

Sounds like something for upstream, i.e. podman vs CNI ?

They have been fighting before, so that's CNI for you...

i.e. when running kubeadm reset wiped the podman network, so containers couldn't run anymore.

I think the previous default was to run them in alphanumerical order, thus the weird "87" prefix ?

For later CRI-O, there should be a configuration entry:

# The default CNI network name to be selected. If not set or "", then
# CRI-O will pick-up the first one found in network_dir.
# cni_default_network = ""

So that can be used to avoid the podman CNI network ?

I don't think that Docker uses CNI, so that's probably why.

[joshmue]

@priyawadhwa @afbjorklund Thank you for your responses!

I don't think that Docker uses CNI, so that's probably why.

That would explain a lot, indeed. I just suspected it, as in the cilium/minikube walkthrough explicitly set --network-plugin=cni, without changing the runtime.

For later CRI-O, there should be a configuration entry:

# The default CNI network name to be selected. If not set or "", then
# CRI-O will pick-up the first one found in network_dir.
# cni_default_network = ""

So that can be used to avoid the podman CNI network ?

I was not able to test this yet, as this apparently did not land in minikube yet. The tweaking of the configuration would be only possible after the cluster started up, right? I think this may lead to pods bound to different networks - just like with the workaround I described in the top. E. g. the control plane's CoreDNS's are provisioned using the default bridge; Workload pods created later get IP's from the custom network plugin.

Perhaps it would be an option to handle 87-podman-bridge.conflist just like 1-k8s.conf which is only created if the --enable-default-cni=true? At least from my (user) perspective, they seem to serve the same purpose, so maybe one of them may be dropped entirely.

IIRC, as result, this would match the behavior of "traditional" kubeadm installations. With kubeadm init being finished, pods which belong to the pod network would not start until a network plugin is running and the K8s node is considered "Ready". At least I also just observed a similar behavior doing some testing in minikube.

Of course, most people most likely just want to get started with minikube as quick as possible, so defaulting --enable-default-cni to true may be best.

Please correct me if I missed anything.

[afbjorklund]

We added the podman CNI so that power users can use podman run (similar to docker run).

It is not needed for Kubernetes, but we are providing both services: Docker/Podman and Kubernetes

It is supposed to be possible to have multiple CNI installed (and configured), without it exploding ?

Anyway, the /etc/cni/net.d config file is not created by kubeadm but just by installing podman.

[tstromberg]

@joshmue - we recently refactored our CNI configurations, but tried to stick as best as possible to preserving existing behaviors. In this case, I think it's possible that we do better now?

Please try:

minikube start --cni=cilium

Please note that I did not change /etc/cni/net.d/87-podman-bridge.conflist.

[joshmue]

@tstromberg Thank you for the information!

I did first quick tests with this flag - without success so far.

I will report back with more qualified feedback ASAP.

[joshmue]

@tstromberg

I was able to do a little more testing now.

When just using minikube start --cni=cilium everything seems to work perfectly. Observation: CoreDNS is recreated when Cilium is ready and the new Pod get's a Cilium IP. kubectl get ciliumendpoints --all-namespaces confirms this. Also, when applying the Cilium connectivity checks in namespace default, everything behaves as expected.

Using minikube start --cni=cilium --container-runtime=crio, the coredns-*** Pod in kube-system keeps getting recreated every 30s once Cilium is ready, but does not seem to obtain an IP address from Cilium. Just 10.244.0.2, then 10.244.0.3 and so on. When applying the Cilium checks, the Pods get IP addresses from the same range. kubectl get ciliumendpoints --all-namespaces is empty in turn. The Pod which does the "denied" check fails and will be restarted from time to time. The other checks seem to work.

[medyagh]

@joshmue thank you for updating the issue, did that answer solve the issue ? or do you think we could still do anything on minikube side to make the experience better on this issue ?

[joshmue]

@joshmue thank you for updating the issue, did that answer solve the issue ? or do you think we could still do anything on minikube side to make the experience better on this issue ?

Hi @medyagh! While the steps written down in my initial comment are now kind of outdated, it would be beneficial to have the --cni=cilium/... flag working without any manual intervention in cases where CRI-O is used. That would improve the experience a lot.

If you want me to update this issue's title or test something, please say so.

[joshmue]

Retested with minikube 1.14.2: Works now without any problems or workarounds. 🎊

Thank you all!