DNS not working in Docker Compose #115

boostpaal · 2018-11-20T17:18:32Z

DNS lookups fail in Docker Compose 2.3.

docker-compose.yml

version: '2.3'
services:
  gvisor_test:
    command: node /home/test.js
    image: node:8-alpine
    runtime: runsc
    volumes:
      - /home/ubuntu/compose/test.js:/home/test.js

test.js

const http = require('http')
http.get('http://www.google.com', res => console.log(res))

Error:

$ docker-compose up
Starting compose_gvisor_test_1 ... done
Attaching to compose_gvisor_test_1
gvisor_test_1  | events.js:183
gvisor_test_1  |       throw er; // Unhandled 'error' event
gvisor_test_1  |       ^
gvisor_test_1  | 
gvisor_test_1  | Error: getaddrinfo EAI_AGAIN www.google.com:80
gvisor_test_1  |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26)
compose_gvisor_test_1 exited with code 1

$ uname -a
Linux ubuntu-2 4.15.0-36-generic #39~16.04.1-Ubuntu SMP Tue Sep 25 08:59:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version:           18.09.0
API version:       1.39
Go version:        go1.10.4
Git commit:        4d60db4
Built:             Wed Nov  7 00:48:57 2018
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
 Version:          18.09.0
 API version:      1.39 (minimum version 1.12)
 Go version:       go1.10.4
 Git commit:       4d60db4
 Built:            Wed Nov  7 00:16:44 2018
 OS/Arch:          linux/amd64
 Experimental:     false

I've tried using the use-vc and other resolve options in Compose (http://man7.org/linux/man-pages/man5/resolv.conf.5.html) to force TCP over UDP, with no luck:

version: '2.3'
services:
  gvisor_test:
    command: node /home/test.js
    image: node:8-alpine
    runtime: runsc
    volumes:
      - /home/ubuntu/compose/test.js:/home/test.js
    dns_opt:
      - use-vc

The text was updated successfully, but these errors were encountered:

iangudger · 2018-12-01T01:23:12Z

Can you post strace logs?
https://github.com/google/gvisor#debugging

boostpaal · 2018-12-01T13:26:42Z

https://gist.github.com/boostpaal/e8a03abdbd3c40ea8b633ec7b2b83873

iangudger · 2018-12-02T05:16:17Z

It appears that you are trying to connect to a DNS server on 127.0.0.11:53, but that there isn't a DNS server in the sandbox running on that port. When it comes to networking, the sandbox behaves like a VM. localhost is the sandbox's localhost. You might be able to talk to the DNS server on the host by using the host's IP address.

Another thing you can try is enabling network passthrough. This makes networking behave more like a normal container at the cost of security. I am not sure if this will help as I think the sandbox runs in a network namespace with its own loopback on the host.

boostpaal · 2018-12-03T10:43:06Z

Network passthrough works but it's not a viable option for security reasons. I seems like Compose's native DNS server is the root of the problem. There might be a missing DNS feature that gVisor requires. I think submitting an issue to Compose would be the right way to go from here.

boostpaal · 2018-12-03T12:55:38Z

The issue can be resolved with a workaround.

Changing the DNS server in Compose is not enough:

version: '2.3'
services:
  apiconbug:
    restart: always
    image: apiconbug
    tty: true
    working_dir: /home/alpine/apiconbug
    runtime: runsc
    dns:
        - 8.8.8.8

However, resolv.conf inside the container would still be:

nameserver 127.0.0.11

which would render the same problem.

When running Compose with runsc, you get two networks:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c9fcf0c8d7dc        bridge              bridge              local
29cf0bbb8e49        nodebug_default     bridge              local

The nodebug_default network is created by Compose when it has runsc as runtime, The Options object is empty:

$ docker network inspect 29cf0bbb8e49
[
    {
        "Name": "apiconbug_default",
        "Id": "a3fcbc8dbf8c8500ff9eee3f4ca7328f7b1dd181ad601168ccbe48be4fee99ac",
        "Created": "2018-11-30T09:50:39.643998227+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},  // <--- EMPTY
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "apiconbug",
            "com.docker.compose.version": "1.22.0"
        }
    }
]

The bridge network has these Options:

$ docker network inspect c9fcf0c8d7dc
[
    {
        "Name": "bridge",
        ...
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        ...
    }
]

We then switched to using the bridge network:

version: '2.3'
services:
  apiconbug:
    restart: always
    image: apiconbug
    tty: true
    working_dir: /home/alpine/apiconbug
    runtime: runsc
    dns:
        - 8.8.8.8
    network_mode: "bridge"

and voila, resolv.conf:

nameserver 8.8.8.8

DNS lookups are now OK.

Would be nice to know which of these options resolve the problem, what the problem actually is, and how to configure these options in Compose.

We could possibly have changed resolv.conf directly, but this file might be changed or overwritten by Docker and is not a viable solution.

This opens access to the hosts' local network, so be sure you know what you are exposing there.

Anjali05 · 2019-05-17T00:09:33Z

I am getting error in apt update inside the ubuntu image when I run it with gvisor, with runc it work fine. Can someone tell me how to fix it?
content of the conf file inside the container:

root@d846d6c54be4:/# cat /etc/resolv.conf
search utah.cloudlab.us
nameserver 10.0.0.2
nameserver 8.8.8.8

error:
root@d846d6c54be4:/# apt update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@d846d6c54be4:/#

jielingit · 2019-06-07T18:09:35Z

I am having the same issue. DNS look up does not work with gVisor and netstack in the Docker container. But without gVisor it works.

I am getting error in apt update inside the ubuntu image when I run it with gvisor, with runc it work fine. Can someone tell me how to fix it?
content of the conf file inside the container:

root@d846d6c54be4:/# cat /etc/resolv.conf
search utah.cloudlab.us
nameserver 10.0.0.2
nameserver 8.8.8.8

error:
root@d846d6c54be4:/# apt update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@d846d6c54be4:/#

iangudger · 2019-06-07T19:22:42Z

I don't think that there is anything we can do to fix this on the gVisor side. Docker Compose seems to be assuming a shared loopback, which I don't think is conceptually compatible with network isolation. I think your only options are to not use network isolation (e.g. enable network passthrough in gVisor) or to fix, not use, or otherwise work around Docker Compose.

jielingit · 2019-06-07T19:53:20Z

I don't think that there is anything we can do to fix this on the gVisor side. Docker Compose seems to be assuming a shared loopback, which I don't think is conceptually compatible with network isolation. I think your only options are to not use network isolation (e.g. enable network passthrough in gVisor) or to fix, not use, or otherwise work around Docker Compose.

I am not using Docker Compose. Just Docker Run. It only has default bridge network. resolv.conf does not point to 127.0.0.11.
network passthrough works. But we would prefer using netstack for network isolation.

docker run --rm -dit --runtime runsc --name ubuntu_runs jie/ubuntu bash
a73e6e85de4cb6337c214f076f454fd5a5f415dc3ef7bd25951709a7a639b370

docker attach a73e6e85de4cb6337c214f076f454fd5a5f415dc3ef7bd25951709a7a639b370
root@a73e6e85de4c:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.21.0.2
search us-west-2.compute.internal
root@a73e6e85de4c:/# 
root@a73e6e85de4c:/# apt update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease                  
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease        
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done        
Building dependency tree       
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@a73e6e85de4c:/# 
root@a73e6e85de4c:/# nc -vz google.com
google.com: forward host lookup failed: Host name lookup failure : Resource temporarily unavailable

iangudger · 2019-06-07T20:10:40Z

That looks like a different problem. Can you file a new issue and attach an strace log?

jielingit · 2019-06-07T20:48:24Z

@iangudger Opened #334

nikosT · 2020-03-30T16:33:25Z

I confirm that DNS name is not working on docker-compose.

D4rk4 · 2020-05-20T11:44:35Z

Still not working with compose. Up!

cuzrawr · 2020-05-22T11:34:06Z

Same issue here

dntq83 · 2020-05-23T01:15:44Z

Same issue

cuzrawr · 2020-05-23T09:44:59Z

You shouldnt edit /etc/resolv.conf in container directly. Leave it 127.0.0.11
And try use this

dns:
    - 8.8.8.8

in docker-compose.yml or *override.yml

D4rk4 · 2020-05-25T09:24:28Z

You shouldnt edit /etc/resolv.conf in container directly. Leave it 127.0.0.11
And try use this
dns:
    - 8.8.8.8
in docker-compose.yml or *override.yml

It's not working for me...

boostpaal · 2020-05-25T11:54:35Z

#115 (comment)

cuzrawr · 2020-05-25T12:18:18Z

#115 (comment)

Its not right when inside container you have external DNS. When you put other address than "127.0.0.11" docker lose self intercommunication between hosts if you are in stack or something.
Docker have own "DNS proxy" which connects from "127.0.0.11" to any external DNS.

Don't put any change in containers resolv.conf. It break functionality. Edit your DNS by other method like dns settings in docker-compose.yml

D4rk4 · 2020-05-25T12:26:14Z

@sloslo4 you right, but anyway dns settings is ignored by docker-compose.

romuloslv · 2020-06-18T01:48:56Z

Still not working, any predictions?

amscanne · 2020-06-18T16:12:54Z

Hopefully someone on the project can look at Docker compose soon, but there's no timeline right now. The problem is fundamentally what Ian notes above. The DNS server appears to be binding to localhost inside the container's network namespace. But the isolated container does not have direct access to the host network namespace unless network=host (passthrough mode) is enabled. For now the workaround suggested above seems reasonable, but I'm sure if there are any consequences wrt to service discovery, etc. (I don't know much about how Docker compose is using DNS and why it is doing what it is doing, which is why we need someone to dig in and understand that first.)

romuloslv · 2020-06-19T16:47:49Z

@amscanne this solution is not working with:

networks:
lab:
ipam:
driver: default
config:
- subnet: 10.98.5.0/16

boredazfcuk · 2020-08-22T18:34:01Z

I'm seeing the same behaviour since I changed from docker run --dns n.n.n.n to docker-compose's dns: n.n.n.n

I have a container that runs VPN, so not being able to specify my VPN provider's DNS servers with dns: in docker-compose means I'm now seeing DNS leaks for VPN traffic, as requests are now hitting the Docker's internal DNS (127.0.0.11:53) which resolves to 8.8.8.8.

ianlewis · 2020-09-08T23:14:59Z

I don't think this is an issue we can solve for good in gVisor as it has to do with how Docker binds to localhost for the DNS server. We can maybe document this better as it seems to be a fairly common issue.

ianlewis · 2020-09-09T05:14:50Z

I added a tutorial to the website that will be available at https://gvisor.dev/docs/tutorials/docker-compose/ so I'm going to close this issue. If there are further issues you have with DNS and docker compose feel free to open up another issue with the details.

Adds a Docker Compose tutorial to the website that shows how to start a Wordpress site and includes information about how to get DNS working. Fixes google#115 PiperOrigin-RevId: 330652842

Adds a Docker Compose tutorial to the website that shows how to start a Wordpress site and includes information about how to get DNS working. Fixes #115 PiperOrigin-RevId: 330652842

mindon · 2021-01-20T10:25:27Z

docker-compose using user defined networks no internal service names or host names resolved

127.0.0.11 names resolve not working (ping ok, nkloopup refused)
/etc/hosts no host entry appended for depends_on services or links alias

docker-compose version 1.27.4, build 40524192

ianlewis · 2021-01-20T23:32:18Z

@mindon This issue is closed. Could you open a new issue with a bit more details about how to reproduce your problem?

mindon · 2021-01-21T07:24:55Z

@mindon This issue is closed. Could you open a new issue with a bit more details about how to reproduce your problem?

my mistake, it should be a docker-compose issue

phirestalker · 2022-11-25T18:55:48Z

Did anyone file a bug with docker-compose or docker about this? If, so would you provide a link to the issue?

boostpaal closed this as completed Nov 20, 2018

boostpaal changed the title ~~Node DNS not working inside docker~~ Node DNS not working inside Docker Compose Nov 30, 2018

boostpaal reopened this Nov 30, 2018

boostpaal changed the title ~~Node DNS not working inside Docker Compose~~ Node DNS not working in Docker Compose Nov 30, 2018

boostpaal changed the title ~~Node DNS not working in Docker Compose~~ DNS not working in Docker Compose Dec 3, 2018

boostpaal closed this as completed Dec 3, 2018

boostpaal reopened this Dec 3, 2018

ianlewis added area: networking Issue related to networking area: container runtime Issue related to docker, kubernetes, OCI runtime labels Jan 17, 2019

Cheaterman mentioned this issue Sep 17, 2019

Version 18.09: local DNS settings are not inherited anymore, local /etc/hosts have no effect inside containers docker/for-linux#488

Open

3 tasks

ianlewis added the area: integration Issue related to third party integrations label Aug 14, 2020

ianlewis added the area: docs Issue related to docs label Sep 8, 2020

copybara-service bot mentioned this issue Sep 9, 2020

Add a Docker Compose tutorial #3868

Merged

copybara-service bot closed this as completed in 00479af Sep 9, 2020

avagin pushed a commit that referenced this issue Sep 10, 2020

Add a Docker Compose tutorial

a29cc27

Adds a Docker Compose tutorial to the website that shows how to start a Wordpress site and includes information about how to get DNS working. Fixes #115 PiperOrigin-RevId: 330652842

pkit mentioned this issue Apr 28, 2022

Netfilter config is not handled on runsc boot, reopen of: "DNS not working in Docker Compose" #7469

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS not working in Docker Compose #115

DNS not working in Docker Compose #115

boostpaal commented Nov 20, 2018 •

edited

Loading

iangudger commented Dec 1, 2018

boostpaal commented Dec 1, 2018

iangudger commented Dec 2, 2018

boostpaal commented Dec 3, 2018

boostpaal commented Dec 3, 2018 •

edited

Loading

Anjali05 commented May 17, 2019

jielingit commented Jun 7, 2019 •

edited

Loading

iangudger commented Jun 7, 2019

jielingit commented Jun 7, 2019 •

edited

Loading

iangudger commented Jun 7, 2019

jielingit commented Jun 7, 2019

nikosT commented Mar 30, 2020

D4rk4 commented May 20, 2020

cuzrawr commented May 22, 2020

dntq83 commented May 23, 2020

cuzrawr commented May 23, 2020

D4rk4 commented May 25, 2020

in docker-compose.yml or *override.yml

boostpaal commented May 25, 2020

cuzrawr commented May 25, 2020

D4rk4 commented May 25, 2020

romuloslv commented Jun 18, 2020

amscanne commented Jun 18, 2020

romuloslv commented Jun 19, 2020

boredazfcuk commented Aug 22, 2020

ianlewis commented Sep 8, 2020

ianlewis commented Sep 9, 2020

mindon commented Jan 20, 2021

ianlewis commented Jan 20, 2021

mindon commented Jan 21, 2021

phirestalker commented Nov 25, 2022

DNS not working in Docker Compose #115

DNS not working in Docker Compose #115

Comments

boostpaal commented Nov 20, 2018 • edited Loading

iangudger commented Dec 1, 2018

boostpaal commented Dec 1, 2018

iangudger commented Dec 2, 2018

boostpaal commented Dec 3, 2018

boostpaal commented Dec 3, 2018 • edited Loading

Anjali05 commented May 17, 2019

jielingit commented Jun 7, 2019 • edited Loading

iangudger commented Jun 7, 2019

jielingit commented Jun 7, 2019 • edited Loading

iangudger commented Jun 7, 2019

jielingit commented Jun 7, 2019

nikosT commented Mar 30, 2020

D4rk4 commented May 20, 2020

cuzrawr commented May 22, 2020

dntq83 commented May 23, 2020

cuzrawr commented May 23, 2020

in docker-compose.yml or *override.yml

D4rk4 commented May 25, 2020

in docker-compose.yml or *override.yml

boostpaal commented May 25, 2020

cuzrawr commented May 25, 2020

D4rk4 commented May 25, 2020

romuloslv commented Jun 18, 2020

amscanne commented Jun 18, 2020

romuloslv commented Jun 19, 2020

boredazfcuk commented Aug 22, 2020

ianlewis commented Sep 8, 2020

ianlewis commented Sep 9, 2020

mindon commented Jan 20, 2021

ianlewis commented Jan 20, 2021

mindon commented Jan 21, 2021

phirestalker commented Nov 25, 2022

boostpaal commented Nov 20, 2018 •

edited

Loading

boostpaal commented Dec 3, 2018 •

edited

Loading

jielingit commented Jun 7, 2019 •

edited

Loading

jielingit commented Jun 7, 2019 •

edited

Loading