Version 18.09: local DNS settings are not inherited anymore, local /etc/hosts have no effect inside containers #488
Comments
|
Hm, I actually think that documentation is incorrect (well, at least partially). I'm also not sure why that worked for you previously in 18.06. Note that on systems that have Docker 18.09 detects if Trying to reproduce the Add an entry to echo "123.123.123.123 foo.bar" >> /etc/hostsPing that host from inside a container (on the default "bridge" network); docker run --rm busybox sh -c 'ping -c1 foo.bar'
ping: bad address 'foo.bar'And the same, using a custom network docker network create bla
docker run --rm --network=bla busybox sh -c 'ping -c1 foo.bar'
ping: bad address 'foo.bar'Adding a custom host to the container works; docker run --rm --network=bla --add-host=foo.bar:123.123.123.123 busybox sh -c 'ping -c1 foo.bar'
PING foo.bar (123.123.123.123): 56 data bytes
--- foo.bar ping statistics ---
1 packets transmitted, 0 packets received, 100% packet lossWhen using the default (bridge) network;
When using "host" networking (
When using a custom bridge network;
The code to generate the initial Example; default bridge networkAdd a comment to the end of echo "# Hello world" >> /etc/resolv.conf
echo "search localdomain.com" >> /etc/resolv.confCheck its content; cat /etc/resolv.conf
nameserver 2001:4860:4860::8844
nameserver 2001:4860:4860::8888
nameserver 8.8.8.8
# Hello world
search localdomain.comNow, start a container on the default (bridge) network, and check the docker run --rm busybox sh -c 'cat /etc/resolv.conf'
nameserver 8.8.8.8
# Hello world
search localdomain.comNow, start a container with a custom DNS set (also on the default "bridge" network); docker run --rm --dns=1.1.1.1 busybox sh -c 'cat /etc/resolv.conf'
search localdomain.com
nameserver 1.1.1.1This time a fresh The custom DNS is written to the container's copy of Example: host networkingIn host networking, the container runs in the host's networking namespace, so will also use the same networking configuration as the host (hence the host's docker run --rm --network=host busybox sh -c 'cat /etc/resolv.conf'
nameserver 2001:4860:4860::8844
nameserver 2001:4860:4860::8888
nameserver 8.8.8.8
# Hello world
search localdomain.comAnd the docker run --rm --network=host busybox sh -c 'cat /etc/hosts'
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
123.123.123.123 foo.bar |
|
So, looks like that part of the docs need some updating;
ping @fcrisciani anything I overlooked here? |
|
@thaJeztah looks good, also if the resolv.conf for some reason cannot be used, like it points to a localhost IP like |
|
I think this might have something to do with whether Ubuntu was installed as 18.04, or whether it was upgraded from 16.04. I came across this same issue when upgrading my Kubernetes hosts from Ubuntu 16.04 and Docker 17.03, to Ubuntu 18.04 and Docker 18.09. The containers all have the same problem where their /etc/resolv.conf points to 8.8.8.8 instead of inheriting. When I downgraded them to Docker 18.06, they work. On the host, I noticed that I also have a separate VM that was installed fresh as Ubuntu 18.04, and that one works fine with Docker 18.09. I also see that the |
|
Can someone from Docker confirm this is an issue? We need to upgrade our Docker hosts to 18.09.2 for the latest runc CVE, but we cannot upgrade without breaking all of our containers. |
|
@shubb30 if you're currently on 18.06, you could still update to 18.06.2 (which has the fix for the CVE). I'm not sure why the |
|
I recently ran into this issue after upgrading our ubuntu 16.04 server to 18.04 and docker to 18.09. originally the dns is set in The proper way to configure I think while reading |
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
Apr 30, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
Apr 30, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
Apr 30, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
Apr 30, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 1, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 2, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.local` to each service in the compose stack, and make sure that `dns_search` for `.local` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .local domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .local, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.local, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 3, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `.internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 3, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `.internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net`, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, so work is ongoing to extricate it. For now, setting the `puppet.conf` values explicitly to the desired DNS name works around this problem as well.
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 3, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases. For now, setting PUPPETSERVER_HOSTNAME configuration value in the puppetserver container will set the `puppet.conf` values explicitly to the desired DNS name to work around this problem. - Resolution of `postgres.internal` seems to rely on having the `hostname` value explicitly defined in the docker-compose file, even though hostname values supposedly don't interact with DNS in docker - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 4, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases. For now, setting PUPPETSERVER_HOSTNAME configuration value in the puppetserver container will set the `puppet.conf` values explicitly to the desired DNS name to work around this problem. - Resolution of `postgres.internal` seems to rely on having the `hostname` value explicitly defined in the docker-compose file, even though hostname values supposedly don't interact with DNS in docker - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 4, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, preventing a need to synchronize a domain name. - Note that at this time there is also a discrepancy in how Facter 3 behaves vs Facter 2. The Facter 2 gem is being used by the `puppetserver ca` gem based application, and may return a different value for Facter.value('domain') than calling `facter domain` at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns the value `delivery.puppetlabs.net` This discrepancy makes it so that the `puppetserver ca` application cannot find the client side cert on disk and fails outright. Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases. For now, setting PUPPETSERVER_HOSTNAME configuration value in the puppetserver container will set the `puppet.conf` values explicitly to the desired DNS name to work around this problem. - Resolution of `postgres.internal` seems to rely on having the `hostname` value explicitly defined in the docker-compose file, even though hostname values supposedly don't interact with DNS in docker - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 6, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, inside of puppet.conf, preventing a need to inject a domain name as was done previously. This is necessary because of a discrepancy in how Facter 3 behaves vs Facter 2, which creates a mismatch between how the host cert is initially generated (using Facter 3) and how `puppetserver ca` finds the files on disk (using Facter 2), that setting PUPPETSERVER_HOSTNAME will explicitly work around. Specifically, Facter 2 may return a different Facter.value('domain') than calling `facter domain` using Facter 3 at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns `delivery.puppetlabs.net` Without explicitly setting PUPPETSERVER_HOSTNAME, this makes cert files on disk get written as *.delivery.puppetlabs.net, yet the `puppetserver ca` application looks for the client certs on disk as *.ops.puppetlabs.net, which causes `puppetserver ca` to fail. - Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases, which may remove the need for the above. - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder
Iristyle
added a commit
to Iristyle/pupperware
that referenced
this issue
May 6, 2019
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f19 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, inside of puppet.conf, preventing a need to inject a domain name as was done previously. This is necessary because of a discrepancy in how Facter 3 behaves vs Facter 2, which creates a mismatch between how the host cert is initially generated (using Facter 3) and how `puppetserver ca` finds the files on disk (using Facter 2), that setting PUPPETSERVER_HOSTNAME will explicitly work around. Specifically, Facter 2 may return a different Facter.value('domain') than calling `facter domain` using Facter 3 at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns `delivery.puppetlabs.net` Without explicitly setting PUPPETSERVER_HOSTNAME, this makes cert files on disk get written as *.delivery.puppetlabs.net, yet the `puppetserver ca` application looks for the client certs on disk as *.ops.puppetlabs.net, which causes `puppetserver ca` to fail. - Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases, which may remove the need for the above. - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder - Additionally update k8s / Bolt specs to be consistent with updated naming
|
This is not good. The least I can confirm is the behavior did change between 17.09.0 and 18.09.1, as the exact same docker-compose configuration works on one and not the other. It seems like the embedded DNS server isn't properly forwarding the requests to the host on the newer version... (EDIT: What makes me think this, is that the resolv.conf in both containers is exactly the same, but one can resolve hostnames and the other cannot) I'd really like a fix that doesn't involve setting the nameserver address manually anywhere since the server acquires it by DHCP (and therefore the address is susceptible to change). EDIT: Problem seems to be solved by adding |
I'm very dubious as to whether this is the right thing to be doing in docker. The # This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.The important part here is "Third party programs must not access this file directly, but only through the symlink at /etc/resolv.conf." If my understanding is correct, In short, it seems to me that the docker container should always inherit from Thoughts? |
|
I totally agree, especially in light of that comment inside the file.
Le mar. 1 oct. 2019 à 21:27, Clayton Lemons <[email protected]> a
écrit :
… @thaJeztah <https://github.com/thaJeztah> @fcrisciani
<https://github.com/fcrisciani>
Note that on systems that have systemd-resolvd enabled, /etc/resolv.conf
is no longer the "leading" configuration file, but another file is used
instead (/run/systemd/resolve/resolv.conf).
I'm very dubious as to whether this is the right thing to be doing in
docker. The "/run/systemd/resolve/resolv.conf" file usually contains the
following comment at the top:
# This file is managed by man:systemd-resolved(8). Do not edit.## This is a dynamic resolv.conf file for connecting local clients directly to# all known uplink DNS servers. This file lists all configured search domains.## Third party programs must not access this file directly, but only through the# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,# replace this symlink by a static file or a different symlink.## See man:systemd-resolved.service(8) for details about the supported modes of# operation for /etc/resolv.conf.
The important part here is *"Third party programs must not access this
file directly, but only through the symlink at /etc/resolv.conf."*
If my understanding is correct, /etc/resolv.conf is the defacto source
for local dns resolution settings (even on ubuntu 18.04+), while
/run/systemd/resolve/resolv.conf is simply an autogenerated file from
systemd-resolvd that *may* be symlinked to. I emphasize "may" because
some third party applications such VPN may choose to ignore systemd's
resolv.conf and instead replace /etc/resolv.conf with their own version
(after backing it up, of course). This scenario is problematic if a docker
container needs to access resources on the VPN but can't because it
inherited from the wrong resolv.conf file. This is a problem I personally
ran into.
In short, it seems to me that the docker container should always inherit
from /etc/resolv.conf, and the user should be responsible for choosing if
/etc/resolv.conf points to /run/systemd/resolve/resolv.conf or contains a
custom configuration.
Thoughts?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#488?email_source=notifications&email_token=AAANLNULQR6IKWK3FVXE6RDQMOQB3A5CNFSM4GDWJYRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEACOILQ#issuecomment-537191470>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAANLNWVJ4RLA5FBH7JDBSTQMOQB3ANCNFSM4GDWJYRA>
.
|
I second that - even if systemd is running the important quote is:
My VPN connection program is replacing that /etc/resolv.conf file and still docker takes the wrong obsolete dns info from the /run/.../resolv.conf file and not the updated /etc/resolv.conf one - DNS obviously does not work anymore inside the container because of that. |
This was referenced Jan 21, 2020
Closed
Closed
|
I confirm that DNS name is not working on docker-compose. |
|
However... yeah.. nevermind. Since the systemd-resolved interface is listening on the loopback interface, that's no good! I really would like to get this working sanely, especially painful on a roadrunner laptop where you move around between nets and vpns alot... This works great... forwarding DNS from docker0 => 127.0.0.53 ... Now the remaining part is influencing all pieces of software that runs docker containers to pass that |
Iristyle
added a commit
to puppetlabs/pupperware
that referenced
this issue
Apr 9, 2021
- Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: docker/for-linux#488 (comment) - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options docker/cli#1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 8b38620 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, inside of puppet.conf, preventing a need to inject a domain name as was done previously. This is necessary because of a discrepancy in how Facter 3 behaves vs Facter 2, which creates a mismatch between how the host cert is initially generated (using Facter 3) and how `puppetserver ca` finds the files on disk (using Facter 2), that setting PUPPETSERVER_HOSTNAME will explicitly work around. Specifically, Facter 2 may return a different Facter.value('domain') than calling `facter domain` using Facter 3 at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns `delivery.puppetlabs.net` Without explicitly setting PUPPETSERVER_HOSTNAME, this makes cert files on disk get written as *.delivery.puppetlabs.net, yet the `puppetserver ca` application looks for the client certs on disk as *.ops.puppetlabs.net, which causes `puppetserver ca` to fail. - Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases, which may remove the need for the above. - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: moby/libnetwork#2371 microsoft/opengcs#303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per docker/for-win#1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder - Additionally update k8s / Bolt specs to be consistent with updated naming
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior
Works in Docker version 18.06.1-ce, build e68fc7a
/etc/hosts file contains some domains for development.
When I ping such domain inside container (inside user defined network) I get the right IP address.
Docs says that container inherits DNS settings from Docker daemon inlcuding /etc/hosts and /etc/resolv.conf. (https://docs.docker.com/config/containers/container-networking/#dns-services)
I don't know exactly how that works but when I inspect the network, this is what happens when pinging domain from /etc/hosts:
Check the image above, you can see that something originating from 127.0.0.1 asks my local DNS resolver at 127.0.0.53, it returns back and then the container seems to already know the address.
Actual behavior
In updated version of docker the DNS query goes from container straight to DNS server outside my network. I know I can override this with --dns option and I do that now because I have no choice. But it was really convenient to just set up /etc/hosts real quick and go.
Steps to reproduce the behavior
Update docker to version 18.09, add record to /etc/hosts locally, ping that domain from container.
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.)
Firewall is up, extra rules are:
22/tcp ALLOW Anywhere
Xdebug ALLOW Anywhere
The text was updated successfully, but these errors were encountered: