docker: pulls may fail if MTU is <1500

[medyagh]

on cloud shell if we dont set this paramter, docker pull on "some images" hangs... (for example on golang:1.15) and not on other images such as alpine ...

sudo sysctl -w net.ipv4.tcp_mtu_probing=1

example:

medya@cloudshell:~$ minikube ssh
docker@minikube:~$ docker pull golang:1.15
^C
docker@minikube:~$ sudo sysctl -w net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_mtu_probing = 1
docker@minikube:~$ docker pull golang:1.15
1.15: Pulling from library/golang
e4c3d3e4f7b0: Pull complete
101c41d0463b: Pull complete
8275efcd805f: Pull complete
751620502a7a: Pull complete
aaabf962c4fc: Pull complete
7883babec904: Pull complete
1791d366c848: Pull complete
Digest: sha256:1ba0da74b20aad52b091877b0e0ece503c563f39e37aa6b0e46777c4d820a2ae
Status: Downloaded newer image for golang:1.15
docker.io/library/golang:1.15

we need to investigate does it make sense to set this by default in our image/iso ?

[medyagh]

this blog post suggests setting the TPU to 1400 would help

https://medium.com/@sylwit/how-we-spent-a-full-day-figuring-out-a-mtu-issue-with-docker-4d81fdfe2caf

for the records our TPUs are 1500

docker@minikube:~$ ip addr | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
4: veth0713d6b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
26: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default

docker allows setting the option for networks

https://docs.docker.com/engine/reference/commandline/network_create/

[tstromberg]

The MTU probing approach can definitely help. MTU's of less than 1500 are kind of exotic in 2020, but are definitely seen within GCP.

One alternative/addition is that the Docker driver could check the MTU of the bridge, and copy it to the custom network it creates.

The upside of enabling probing is that it may also improve the situation for non-Docker drivers, although our scripts for setting sysctl settings differ between ISO & containers.