From 065352876bd578e566bdf19eb2c3dc3cc62f061f Mon Sep 17 00:00:00 2001 From: Iristyle Date: Wed, 24 Apr 2019 15:55:29 -0700 Subject: [PATCH 1/3] (maint) Add puppetserver alias puppet.internal - Remove the domain introspection / setting of AZURE_DOMAIN env var as this does not work as originally thought. Instead, hardcode the DNS suffix `.internal` to each service in the compose stack, and make sure that `dns_search` for `internal` will use the Docker DNS resolver when dealing with these hosts. Note that these compose file settings only affect the configuration of the DNS resolver, *not* resolv.conf. This is different from the docker run behavior, which *does* modify resolv.conf. Also note, config file locations vary depending on whether or not systemd is running in the container. It's not "safe" to refer to services in the cluster by only their short service names like `puppet`, `puppetdb` or `postgres` as they can conflict with hosts on the external network with these names when `resolv.conf` appends DNS search suffixes. When docker compose creates the user defined network, it copies the DNS settings from the host to the `resolv.conf` in each of the containers. This often takes search domains from the outside network and applies them to containers. When network resolutions happen, any default search suffix will be applied to short names when the dns option for ndots is not set to 0. So for instance, given a `resolv.conf` that contains: search delivery.puppetlabs.net A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net` which will fail to resolve in the Docker DNS resolver, then be sent to the next DNS server in the `nameserver` list, which may resolve it to a different host in the external network. This behaves this way because `resolv.conf` also sets secondary DNS servers from the host. While it is possible to try and service requests for an external domain like `delivery.puppetlabs.net` with the embedded Docker DNS resolver, it's better to instead choose a domain suffix to use inside the cluster. There are some good details on how various network types configure: https://github.com/docker/for-linux/issues/488#issuecomment-438830363 - Note that the .internal domain is typically not recommended for production given the only IANA reserved domains are .example, .test, .invalid or .localhost. However, given the DNS resolver is set to own the resolution of .internal, this is a compromise. In production its recommended to use a subdomain of a domain that you own, but that's not yet configurable in this compose file. A future commit will make this configurable. - Another workaround for this problem would be to set the ndots option in resolv.conf to 0 per the documentation at http://man7.org/linux/man-pages/man5/resolv.conf.5.html However that can't be done for two reasons: - docker-compose schema doesn't actually support setting DNS options https://github.com/docker/cli/issues/1557 - k8s sets ndots to 5 by default, so we don't want to be at odds - A further, but implausible workaround would be to modify the host DNS settings to remove any search suffixes. - The original FQDN change being reverted in this commit was introduced in 2549f1970b23d890baa3cfe1fc8eb5b3e6de67d8 " Lastly, the Windows specific docker-compose.windows.yml sets up a custom alias in the "default" network so that an extra DNS name for puppetserver can be set based on the FQDN that Facter determines. Without this additional DNS reservation, the `puppetserver ca` command will be unable to connect to the REST endpoint. A better long-term solution is making sure puppetserver is setup to point to `puppet` as the host instead of an FQDN. " With the PUPPETSERVER_HOSTNAME value set on the puppetserver container, both certname and server are set to puppet.internal, inside of puppet.conf, preventing a need to inject a domain name as was done previously. This is necessary because of a discrepancy in how Facter 3 behaves vs Facter 2, which creates a mismatch between how the host cert is initially generated (using Facter 3) and how `puppetserver ca` finds the files on disk (using Facter 2), that setting PUPPETSERVER_HOSTNAME will explicitly work around. Specifically, Facter 2 may return a different Facter.value('domain') than calling `facter domain` using Facter 3 at the command line. Such is the case inside the puppet network, where Facter 2 returns `ops.puppetlabs.net` while Facter 3 returns `delivery.puppetlabs.net` Without explicitly setting PUPPETSERVER_HOSTNAME, this makes cert files on disk get written as *.delivery.puppetlabs.net, yet the `puppetserver ca` application looks for the client certs on disk as *.ops.puppetlabs.net, which causes `puppetserver ca` to fail. - Facter 2 should not be included in the puppetserver packages, and changes have been made to packaging for future releases, which may remove the need for the above. - This PR is also made possible by switching over to using the Ubuntu based container from the Alpine container (performed in a prior commit), due to DNS resolution problems with Alpine inside LCOW: https://github.com/docker/libnetwork/issues/2371 https://github.com/Microsoft/opengcs/issues/303 - Another avenue that was investigated to resolve the DNS problem in Alpine was to feed host:ip mappings in through --add-host, but it turns out that Windows doesn't yet support that feature per https://github.com/docker/for-win/issues/1455 - Finally, these changes are also made in preparation of switching the pupperware-commercial repo over to a private builder - Additionally update k8s / Bolt specs to be consistent with updated naming --- Boltdir/modules/stack/tasks/manage.sh | 4 ++-- Boltdir/modules/stack/tasks/pdb_node.sh | 2 +- README.md | 12 +++++----- azure-pipelines.yml | 4 ---- docker-compose.yml | 29 +++++++++++++++++++------ gem/lib/pupperware/spec_helper.rb | 12 +++++----- k8s/README.md | 2 +- k8s/bin/puppet-query | 2 +- k8s/postgres.yaml | 1 + k8s/puppetdb.yaml | 6 ++++- k8s/puppetserver.yaml | 10 ++++++--- spec/dockerfile_spec.rb | 4 +++- 12 files changed, 54 insertions(+), 34 deletions(-) diff --git a/Boltdir/modules/stack/tasks/manage.sh b/Boltdir/modules/stack/tasks/manage.sh index 5970f8a1..8721bede 100644 --- a/Boltdir/modules/stack/tasks/manage.sh +++ b/Boltdir/modules/stack/tasks/manage.sh @@ -7,7 +7,7 @@ die() { pdb_running() { docker-compose exec -T puppet \ - curl -s 'http://puppetdb:8080/status/v1/services/puppetdb-status' | \ + curl -s 'http://puppetdb.internal:8080/status/v1/services/puppetdb-status' | \ python -c 'import json, sys; print json.load(sys.stdin)["state"]' } @@ -27,7 +27,7 @@ wait_for_it() { cd pupperware || die no-repo "run the clone task first to set up pupperware" host=$(getent hosts "$(hostname -s)") -export DNS_ALT_NAMES="puppet,${host##* }" +export DNS_ALT_NAMES="puppet,puppet.internal,${host##* }" case $PT_action in up) diff --git a/Boltdir/modules/stack/tasks/pdb_node.sh b/Boltdir/modules/stack/tasks/pdb_node.sh index 7ae91252..c0e9c2c3 100644 --- a/Boltdir/modules/stack/tasks/pdb_node.sh +++ b/Boltdir/modules/stack/tasks/pdb_node.sh @@ -7,7 +7,7 @@ body=$(printf '{ "query": "nodes { certname = \\"%s\\" }" }' "$PT_agent") out="" while [ -z "$out" ]; do out=$(docker-compose exec -T puppet \ - curl -s -X POST http://puppetdb:8080/pdb/query/v4 \ + curl -s -X POST http://puppetdb.internal:8080/pdb/query/v4 \ -H 'Content-Type:application/json' \ -d "$body") if [ -z "$out" ]; then diff --git a/README.md b/README.md index dd58af19..20cf9bea 100644 --- a/README.md +++ b/README.md @@ -31,12 +31,12 @@ Once you have Docker Compose installed, you can start the stack on Linux with: The value of `DNS_ALT_NAMES` must list all the names, as a comma-separated list, under which the Puppet server in the stack can be reached from -agents. It will have `puppet` prepended to it as that name is used by PuppetDB -to communicate with the Puppet server. The value of `DNS_ALT_NAMES` only has an -effect the first time you start the stack, as it is placed into the server's SSL -certificate. If you need to change it after that, you will need to properly -revoke the server's certificate and restart the stack with the changed -`DNS_ALT_NAMES` value. +agents. It will have `puppet` and `puppet.internal` prepended to it as that +name is used by PuppetDB to communicate with the Puppet server. The value of +`DNS_ALT_NAMES` only has an effect the first time you start the stack, as it +is placed into the server's SSL certificate. If you need to change it after +that, you will need to properly revoke the server's certificate and restart +the stack with the changed `DNS_ALT_NAMES` value. When you first start the Puppet Infrastructure, the stack will create a `volumes/` directory with a number of sub-directories to store the diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 9bae94ec..8a2971a5 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -65,12 +65,8 @@ steps: name: test_prepare - powershell: | - $domain = Get-WmiObject -Class Win32_NetworkAdapterConfiguration | - Select -ExpandProperty DNSDomain | - Select -First 1 Write-Host 'Writing compose config to disk' $content = @" - AZURE_DOMAIN=$domain VOLUME_ROOT=$ENV:TempVolumeRoot "@ $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False diff --git a/docker-compose.yml b/docker-compose.yml index 392f1597..efae8803 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,26 +2,30 @@ version: '3' services: puppet: - hostname: puppet + hostname: puppet.internal image: puppet/puppetserver ports: - 8140:8140 environment: + # necessary to set certname and server in puppet.conf, required by + # puppetserver ca cli application + - PUPPETSERVER_HOSTNAME=puppet.internal # DNS_ALT_NAMES must be set before starting the stack the first time, # and must list all the names under which the puppetserver can be - # reached. 'puppet' must be one of them, otherwise puppetdb won't be + # reached. 'puppet.internal' must be one of them, otherwise puppetdb won't be # able to get a cert. Add other names as a comma-separated list - - DNS_ALT_NAMES=puppet,${DNS_ALT_NAMES:-} + - DNS_ALT_NAMES=puppet,puppet.internal,${DNS_ALT_NAMES:-} - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true} - - PUPPETDB_SERVER_URLS=https://puppetdb:8081 + - PUPPETDB_SERVER_URLS=https://puppetdb.internal:8081 volumes: - ${VOLUME_ROOT:-.}/volumes/code:/etc/puppetlabs/code/ - ${VOLUME_ROOT:-.}/volumes/puppet:/etc/puppetlabs/puppet/ - ${VOLUME_ROOT:-.}/volumes/serverdata:/opt/puppetlabs/server/data/puppetserver/ + dns_search: internal networks: default: aliases: - - puppet.${AZURE_DOMAIN:-} + - puppet.internal postgres: image: postgres:9.6 @@ -34,14 +38,20 @@ services: volumes: - ${VOLUME_ROOT:-.}/volumes/puppetdb-postgres/data:/var/lib/postgresql/data - ./postgres-custom:/docker-entrypoint-initdb.d + dns_search: internal + networks: + default: + aliases: + - postgres.internal puppetdb: - hostname: puppetdb + hostname: puppetdb.internal image: puppet/puppetdb environment: - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true} # This name is an FQDN so the short name puppet doesn't collide outside compose network - - PUPPETSERVER_HOSTNAME=puppet.${AZURE_DOMAIN:-} + - PUPPETSERVER_HOSTNAME=puppet.internal + - PUPPETDB_POSTGRES_HOSTNAME=postgres.internal - PUPPETDB_PASSWORD=puppetdb - PUPPETDB_USER=puppetdb ports: @@ -52,3 +62,8 @@ services: - puppet volumes: - ${VOLUME_ROOT:-.}/volumes/puppetdb/ssl:/etc/puppetlabs/puppet/ssl/ + dns_search: internal + networks: + default: + aliases: + - puppetdb.internal diff --git a/gem/lib/pupperware/spec_helper.rb b/gem/lib/pupperware/spec_helper.rb index 2d022f24..57e51193 100644 --- a/gem/lib/pupperware/spec_helper.rb +++ b/gem/lib/pupperware/spec_helper.rb @@ -225,11 +225,10 @@ def wait_on_puppetserver_status(seconds = 180, service_name = 'puppet') end end + # agent_name is the fully qualified name of the node def clean_certificate(agent_name) - result = run_command('docker-compose --no-ansi exec -T puppet facter domain') - domain = result[:stdout].chomp - STDOUT.puts "cleaning cert for #{agent_name}.#{domain}" - result = run_command("docker-compose --no-ansi exec -T puppet puppetserver ca clean --certname #{agent_name}.#{domain}") + STDOUT.puts "cleaning cert for #{agent_name}" + result = run_command("docker-compose --no-ansi exec -T puppet puppetserver ca clean --certname #{agent_name}") return result[:status].exitstatus end @@ -253,11 +252,10 @@ def run_agent(agent_name, network, server = get_container_hostname(get_service_c return result[:status].exitstatus end + # agent_name is the fully qualified name of the node def check_report(agent_name) pdb_uri = URI::join(get_service_base_uri('puppetdb', 8080), '/pdb/query/v4') - result = run_command("docker-compose --no-ansi exec -T puppet facter domain") - domain = result[:stdout].chomp - body = "{ \"query\": \"nodes { certname = \\\"#{agent_name}.#{domain}\\\" } \" }" + body = "{ \"query\": \"nodes { certname = \\\"#{agent_name}\\\" } \" }" return retry_block_up_to_timeout(120) do Net::HTTP.start(pdb_uri.hostname, pdb_uri.port) do |http| diff --git a/k8s/README.md b/k8s/README.md index 2df0e874..72866467 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -14,7 +14,7 @@ running Kubernetes via Docker for Mac, this will be the FQDN of your Mac. Note t ```yaml - name: DNS_ALT_NAMES - value: puppet,myworkstation.domain.net + value: puppet,puppet.internal,myworkstation.domain.net ``` Then create the Pupperware resources: diff --git a/k8s/bin/puppet-query b/k8s/bin/puppet-query index d3416c99..6bdcf7e1 100755 --- a/k8s/bin/puppet-query +++ b/k8s/bin/puppet-query @@ -1,3 +1,3 @@ #! /bin/sh -kubectl get pods --selector=svc=puppetdb -o name | cut -d '/' -f 2 | xargs -I '%' kubectl exec '%' -- curl -s -X GET http://puppetdb:8080/pdb/query/v4 --data-urlencode "query=$@" +kubectl get pods --selector=svc=puppetdb -o name | cut -d '/' -f 2 | xargs -I '%' kubectl exec '%' -- curl -s -X GET http://puppetdb.internal:8080/pdb/query/v4 --data-urlencode "query=$@" diff --git a/k8s/postgres.yaml b/k8s/postgres.yaml index 74414c95..539c39e2 100644 --- a/k8s/postgres.yaml +++ b/k8s/postgres.yaml @@ -44,6 +44,7 @@ spec: app: pupperware svc: postgres spec: + hostname: postgres.internal containers: - image: puppet/puppetdb-postgres name: postgres diff --git a/k8s/puppetdb.yaml b/k8s/puppetdb.yaml index fc2bb7c5..28b9cc4f 100644 --- a/k8s/puppetdb.yaml +++ b/k8s/puppetdb.yaml @@ -47,11 +47,15 @@ spec: app: pupperware svc: puppetdb spec: - hostname: puppetdb + hostname: puppetdb.internal containers: - image: puppet/puppetdb name: puppetdb env: + - name: PUPPETSERVER_HOSTNAME + value: puppet.internal + - name: PUPPETDB_POSTGRES_HOSTNAME + value: postgres.internal - name: PUPPETDB_PASSWORD valueFrom: secretKeyRef: diff --git a/k8s/puppetserver.yaml b/k8s/puppetserver.yaml index 843a7942..37d7dc3f 100644 --- a/k8s/puppetserver.yaml +++ b/k8s/puppetserver.yaml @@ -71,15 +71,19 @@ spec: app: pupperware svc: puppet spec: - hostname: puppet + hostname: puppet.internal containers: - image: puppet/puppetserver name: puppet env: + # necessary to set certname and server in puppet.conf, required by + # puppetserver ca cli application + - name: PUPPETSERVER_HOSTNAME + value: puppet.internal - name: DNS_ALT_NAMES - value: puppet + value: puppet,puppet.internal - name: PUPPETDB_SERVER_URLS - value: https://puppetdb:8081 + value: https://puppetdb.internal:8081 ports: - containerPort: 8140 volumeMounts: diff --git a/spec/dockerfile_spec.rb b/spec/dockerfile_spec.rb index b9c91999..c56faa82 100644 --- a/spec/dockerfile_spec.rb +++ b/spec/dockerfile_spec.rb @@ -14,7 +14,9 @@ ] before(:all) do - @test_agent = "puppet_test#{Random.rand(1000)}" + # append .internal to ensure domain suffix for Docker DNS resolver is used + # since search domains are not appended to /etc/resolv.conf + @test_agent = "puppet_test#{Random.rand(1000)}.internal" @timestamps = [] status = run_command('docker-compose --no-ansi version')[:status] if status.exitstatus != 0 From 756dcdc36aefefa44352d3f908a6dcb8c229312b Mon Sep 17 00:00:00 2001 From: Iristyle Date: Fri, 3 May 2019 16:43:21 -0700 Subject: [PATCH 2/3] (maint) Add get_container_ip spec helper - While this is not used anywhere (yet?), make it a helper because it took a little bit of effort to figure out how to extract this information --- gem/lib/pupperware/spec_helper.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/gem/lib/pupperware/spec_helper.rb b/gem/lib/pupperware/spec_helper.rb index 57e51193..7be025ba 100644 --- a/gem/lib/pupperware/spec_helper.rb +++ b/gem/lib/pupperware/spec_helper.rb @@ -137,6 +137,11 @@ def get_container_hostname(container) return fqdn || inspect_container(container, '{{.Config.Hostname}}') end + # this only works when a container has a single network + def get_container_ip(container) + inspect_container(container, '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') + end + def emit_log(container) container_name = get_container_name(container) STDOUT.puts("#{'*' * 80}\nContainer logs for #{container_name} / #{container}\n#{'*' * 80}\n") From 8c1cdae21f641512f41aa5dd254e6faf56b4f2c6 Mon Sep 17 00:00:00 2001 From: Iristyle Date: Fri, 3 May 2019 16:55:32 -0700 Subject: [PATCH 3/3] (maint) Allow .internal domain to be configurable - Modify compose to use either the ENV variable DOMAIN supplied by an end user or to fall back to `internal` when its left unspecified. --- README.md | 9 +++++++++ docker-compose.yml | 28 ++++++++++++++-------------- spec/dockerfile_spec.rb | 4 ++-- 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 20cf9bea..19270bbe 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,15 @@ is placed into the server's SSL certificate. If you need to change it after that, you will need to properly revoke the server's certificate and restart the stack with the changed `DNS_ALT_NAMES` value. +Optionally, you may also provide a desired `DOMAIN` value, other than default +value of `internal` to further define how the service hosts are named. It is +not necessary to change `DNS_ALT_NAMES` as the default value already takes into +account any custom domain. + +``` + DOMAIN=foo docker-compose up -d +``` + When you first start the Puppet Infrastructure, the stack will create a `volumes/` directory with a number of sub-directories to store the persistent data that should survive the restart of your infrastructure. This diff --git a/docker-compose.yml b/docker-compose.yml index efae8803..dd7420e5 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,30 +2,30 @@ version: '3' services: puppet: - hostname: puppet.internal + hostname: puppet.${DOMAIN:-internal} image: puppet/puppetserver ports: - 8140:8140 environment: # necessary to set certname and server in puppet.conf, required by # puppetserver ca cli application - - PUPPETSERVER_HOSTNAME=puppet.internal + - PUPPETSERVER_HOSTNAME=puppet.${DOMAIN:-internal} # DNS_ALT_NAMES must be set before starting the stack the first time, # and must list all the names under which the puppetserver can be - # reached. 'puppet.internal' must be one of them, otherwise puppetdb won't be + # reached. 'puppet.${DOMAIN:-internal}' must be one of them, otherwise puppetdb won't be # able to get a cert. Add other names as a comma-separated list - - DNS_ALT_NAMES=puppet,puppet.internal,${DNS_ALT_NAMES:-} + - DNS_ALT_NAMES=puppet,puppet.${DOMAIN:-internal},${DNS_ALT_NAMES:-} - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true} - - PUPPETDB_SERVER_URLS=https://puppetdb.internal:8081 + - PUPPETDB_SERVER_URLS=https://puppetdb.${DOMAIN:-internal}:8081 volumes: - ${VOLUME_ROOT:-.}/volumes/code:/etc/puppetlabs/code/ - ${VOLUME_ROOT:-.}/volumes/puppet:/etc/puppetlabs/puppet/ - ${VOLUME_ROOT:-.}/volumes/serverdata:/opt/puppetlabs/server/data/puppetserver/ - dns_search: internal + dns_search: ${DOMAIN:-internal} networks: default: aliases: - - puppet.internal + - puppet.${DOMAIN:-internal} postgres: image: postgres:9.6 @@ -38,20 +38,20 @@ services: volumes: - ${VOLUME_ROOT:-.}/volumes/puppetdb-postgres/data:/var/lib/postgresql/data - ./postgres-custom:/docker-entrypoint-initdb.d - dns_search: internal + dns_search: ${DOMAIN:-internal} networks: default: aliases: - - postgres.internal + - postgres.${DOMAIN:-internal} puppetdb: - hostname: puppetdb.internal + hostname: puppetdb.${DOMAIN:-internal} image: puppet/puppetdb environment: - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true} # This name is an FQDN so the short name puppet doesn't collide outside compose network - - PUPPETSERVER_HOSTNAME=puppet.internal - - PUPPETDB_POSTGRES_HOSTNAME=postgres.internal + - PUPPETSERVER_HOSTNAME=puppet.${DOMAIN:-internal} + - PUPPETDB_POSTGRES_HOSTNAME=postgres.${DOMAIN:-internal} - PUPPETDB_PASSWORD=puppetdb - PUPPETDB_USER=puppetdb ports: @@ -62,8 +62,8 @@ services: - puppet volumes: - ${VOLUME_ROOT:-.}/volumes/puppetdb/ssl:/etc/puppetlabs/puppet/ssl/ - dns_search: internal + dns_search: ${DOMAIN:-internal} networks: default: aliases: - - puppetdb.internal + - puppetdb.${DOMAIN:-internal} diff --git a/spec/dockerfile_spec.rb b/spec/dockerfile_spec.rb index c56faa82..e3d23a56 100644 --- a/spec/dockerfile_spec.rb +++ b/spec/dockerfile_spec.rb @@ -14,9 +14,9 @@ ] before(:all) do - # append .internal to ensure domain suffix for Docker DNS resolver is used + # append .internal (or user domain) to ensure domain suffix for Docker DNS resolver is used # since search domains are not appended to /etc/resolv.conf - @test_agent = "puppet_test#{Random.rand(1000)}.internal" + @test_agent = "puppet_test#{Random.rand(1000)}.#{ENV['DOMAIN'] || 'internal'}" @timestamps = [] status = run_command('docker-compose --no-ansi version')[:status] if status.exitstatus != 0