containerd-1.6.6-4.ph4.x86_64 breaks the Docker container runtime

[aad-griff]

Describe the bug

The latest containerd package (1.6.6-4.ph4.x86_64) appears to break networking in some containers.
When testing this package with couchdb 2.3.0 the container exhibited a large consumption of RAM and never opens the default TCP listen port 5984.

Reproduction steps

A fresh install of Photon 4.0 starts with containerd-1.4.4-8.ph4.x86_64 - all containers work as expected using this package.
After performing a tdnf update, the following packages are installed (they all depend on each other for installation):

• containerd-extras-1.6.6-4.ph4
• containerd-1.6.6-4.ph4
• runc-1.1.4-2.ph4

After rebooting Photon 4, run the following command to test a couchdb container:

systemctl start docker && CONTAINER=$(docker run -d --rm couchdb:2.3.0) && docker exec -it ${CONTAINER} /bin/bash -c "apt update ; apt -y install net-tools && sleep 5 && netstat -pantu" && docker rm -f ${CONTAINER}

Note that the couchdb process never starts listening on the default port 5984 (as would be expected). Reverting the packages back to the following versions resolves the issue after a reboot:

• containerd-1.4.4-8.ph4.x86_64
• containerd-extras-1.4.4-8.ph4.x86_64
• runc-1.0.0.rc93-5.ph4.x86_64

Expected behavior

Couchdb containers should begin to listen on port 5984 after a tdnf update.

Additional context

No response

[sshedi]

Try it manually. I don't think there is network issue. Here is what I did:

docker run -it --rm couchdb:2.3.0 /bin/bash
apt update; apt install -y iputils-ping net-tools
ping google.com -------> working fine, hence no network issue

Something else is going wrong.

[aad-griff]

It's important to install the containerd-1.6.6-4.ph4.x86_64 and reboot the VMware Photon guest OS before testing.

After more testing I have discovered that I can only reproduce this issue on a VMware hypervisor specifically with the "VMXNET 3" network adapter! (changing to E1000E resolves the issue)

When running the container manually, netstat should report port 5984 listening:

docker run -it --rm couchdb:2.3.0 /bin/bash;
apt update;
apt install -y net-tools;
/docker-entrypoint.sh /opt/couchdb/bin/couchdb &
sleep 10; # Wait for couchdb to start and initialise a listening port
netstat -pantu # This should list port 5984 as listening but it doesn't start listening at all!

VMWare Environment:
Version: 7.0.3
Build: 20395099

[sshedi]

The issue is reproducible in fedora as well. I have raised an issue in CouchDB upstream, can you please follow up there? They suggested to use a newer image, try with it and give an update.

[sshedi]

cc: @YustasSwamp @srivatsabhat

[aad-griff]

I have verified that testing with couchdb:3.2.2 (the most recent version) does not exhibit this issue however all of the other versions 2.3.1 and 3.1.2 do not work. It concerns that this containerd update in combination with the VMXNET 3 ethernet adapter may have broader implications for other (non-couchdb) containers running on VMWare hypervisors as well...