user-v2 regression in master: DNS wouldn't work with existing VMs

[AkihiroSuda]

@balajiv113: I found that after upgrading from lima to 0.17.2 to HEAD-de1b3ee, DNS wouldn't work with existing VMs. I had to delete and recreate the VM for DNS to work at all. Is that expected?

Originally posted by @aaronlehmann in #1333 (comment)

[balajiv113]

I checked this. Iptables looks fine.

But the nameservers in user-data is not updated to new record after new version update

[balajiv113]

I couldn't find a way to fix this yet. One thing i noticed is the dns nameserver not updated correctly. Apart from that all were looking fine only.

Let me know if we have some ways to enforce cloud-init to update nameservers on every boot

[AkihiroSuda]

If the issue happens only on upgrade we can just let hostagent fix up DNS?

[balajiv113]

@AkihiroSuda
This is actually not happening.

Started net-user-v2 in old version. Did stop on old version. Started the same in current master DNS is working fine.

There are few other issue but unrelated to regression

[AkihiroSuda]

Thank you, seems we can close this

[aaronlehmann]

I could definitely reproduce this reliably when I reported the issue. I would create a VM with colima, upgrade lima to HEAD, and always get broken DNS. I didn't have this problem when I started with lima HEAD. I'm traveling right now but would be happy to provide more details when I get back.

[aaronlehmann]

@AkihiroSuda @balajiv113: Repro steps:

➜  ~ colima delete
are you sure you want to delete colima and all settings? [y/N] y
INFO[0001] deleting colima
INFO[0001] done
➜  ~ brew uninstall --ignore-dependencies lima
Uninstalling /opt/homebrew/Cellar/lima/HEAD-de1b3ee... (111 files, 169.3MB)
lima 0.17.2 is still installed.
To remove all versions, run:
  brew uninstall --force lima
➜  ~ brew link lima
Linking /opt/homebrew/Cellar/lima/0.17.2... 43 symlinks created.
➜  ~ lima -v
limactl version 0.17.2
➜  ~ colima start
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] creating and starting ...                     context=vm
INFO[0021] provisioning ...                              context=docker
INFO[0021] starting ...                                  context=docker
INFO[0026] done
➜  ~ docker run -ti --rm alpine ping google.com
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
9fda8d8052c6: Pull complete
Digest: sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
Status: Downloaded newer image for alpine:latest
PING google.com (142.251.214.142): 56 data bytes
64 bytes from 142.251.214.142: seq=0 ttl=63 time=0.760 ms
64 bytes from 142.251.214.142: seq=1 ttl=63 time=0.980 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.760/0.870/0.980 ms
➜  ~ colima stop
INFO[0000] stopping colima
INFO[0000] stopping ...                                  context=docker
INFO[0001] stopping ...                                  context=vm
INFO[0004] done
➜  ~ brew unlink lima
Unlinking /opt/homebrew/Cellar/lima/0.17.2... 43 symlinks removed.
➜  ~ brew install --head lima
[...]
➜  ~ lima -v
limactl version HEAD-572b94a
➜  ~ docker run -ti --rm alpine ping google.com
ping: bad address 'google.com'

[balajiv113]

I don't think this has anything to do with user-v2 mode. But i will check on these following steps and see if its failing

[balajiv113]

@aaronlehmann could you try directly with lima ?? To see if that works.

As far as i know colima adds their own dns resolver as well for qemu, this might be something related to that as well

[balajiv113]

@aaronlehmann Thanks for the bug. I was able to re-produce it with alpine image

@AkihiroSuda Looks like if a nameserver gets updated in cloud-init that's not reflected in /etc/resolv.conf on next start

[balajiv113]

It works if i do rc-service networking restart

[AkihiroSuda]

Should have been solved in:

• Fix DNS issue with alpine after upgrade #1898

[aaronlehmann]

Confirmed the fix, thanks!