Support Puppet 7 #779

ekohl · 2021-03-08T17:17:32Z

https://puppet.com/docs/puppet/7.4/server/release_notes.html

Handle cadir move to /etc/puppetlabs/puppetserver/ca
Handle auth.conf removal
Handle JSON fact cache instead of YAML (as used by Foreman's ENC)
Look at any legacy Puppet 3 endpoints
Align default ciphers

mattock · 2021-03-24T13:33:29Z

@ekohl I'm working on this as well. I've issued a few rather trivial metadata.json PRs which make Kafo happy about Puppet version requirements.

I was able to run our own puppetmaster-installer fine, but it fails with authorization errors. I'll look into that more tomorrow.

If you have anything work in progress / unpushed let me know so that we can coordinate the work 😄.

ekohl · 2021-03-24T13:46:50Z

I haven't had time so all my work is here. Fact uploading is certainly broken right now. I do have some code for it and started theforeman/puppet-foreman#925 but puppetserver isn't writing out the files on my server and I couldn't figure it out until now.

ekohl · 2021-03-24T13:47:34Z

Also linking it back here: https://community.theforeman.org/t/puppet-7-support/22786

manifests/server/puppetserver.pp

ekohl · 2021-06-04T15:10:52Z

I pushed the cadir handling, but haven't had a chance to test it. That's why it's still a draft and not checked off.

manifests/server/config.pp

ekohl · 2021-07-23T15:17:57Z

I believe this is now ready. Let's see if the tests agree.

ekohl · 2021-07-23T15:37:24Z

spec/acceptance/puppetserver_upgrade_spec.rb

+  case ENV['BEAKER_PUPPET_COLLECTION']
+  when 'puppet7'
+    from_version = '7.0.0'
+    to_version = '7.2.0'
+  when 'puppet6'
+    from_version = '6.7.0'
+    to_version = '6.7.2'
+  when 'puppet5'
+    from_version = '5.3.6'
+    to_version = '5.3.7'


Somehow this finds puppet6 on Debian builds but not on CentOS. I don't understand why yet. There is a default is set here so perhaps it's not propagating somehow.

ehelms · 2021-07-23T17:19:19Z

spec/acceptance/puppetserver_upgrade_spec.rb

-    from_version = "5.3.6-1#{fact('lsbdistcodename')}"
-    to_version = "5.3.7-1#{fact('lsbdistcodename')}"
+    from_version_exact = "6.7.0-1#{fact('lsbdistcodename')}"
+    to_version_exact = "6.7.2-1#{fact('lsbdistcodename')}"


@ekohl Is this not the reason it is finding puppet6 on Debian? As this code says, when its Debian always use these versions.

That explains it. I'm not sure how I missed that when writing it.

ekohl · 2021-07-25T16:39:39Z

I think the max open file limit is failing on Puppet 5 because it's pulling in camptocamp/systemd 3 (bd359f0) which has dropped support for Puppet 5. However, it is odd that it's failing here and not in master.

This was never built so there's nothing to test

This was already true for AIO packages, but at least in Debian 10 and Fedora 33's non-AIO packages this is also true.

In Puppetserver 7 the legacy auth.conf has been removed. This also means the use-legacy-auth-conf setting is useless.

These are present in the default Puppetserver 7 auth.conf.

ekohl · 2021-07-26T14:27:17Z

CentOS 6 build failures are expected, just like Puppet 5 + Ubuntu 20.04. Rest is still running. If they're green, I think this should be merged.

ehelms · 2021-07-26T14:34:24Z

Looks like down to just CI / Acceptance / Puppet 7 - Ubuntu 20.04 (pull_request)

ekohl · 2021-07-26T15:41:54Z

Looks like this is repeatable. The debug log:

  Notice: /Stage[main]/Puppet::Server::Puppetserver/Systemd::Dropin_file[puppetserver.service-limits.conf]/File[/etc/systemd/system/puppetserver.service.d]/ensure: created
  Debug: /Stage[main]/Puppet::Server::Puppetserver/Systemd::Dropin_file[puppetserver.service-limits.conf]/File[/etc/systemd/system/puppetserver.service.d]: The container Systemd::Dropin_file[puppetserver.service-limits.conf] will propagate my refresh event
  Notice: /Stage[main]/Puppet::Server::Puppetserver/Systemd::Dropin_file[puppetserver.service-limits.conf]/File[/etc/systemd/system/puppetserver.service.d/limits.conf]/ensure: defined content as '{sha256}af70de30e188923816fe1eb77708cdc603f53918edd1f7815bea5bc48690f062'
  Debug: /Stage[main]/Puppet::Server::Puppetserver/Systemd::Dropin_file[puppetserver.service-limits.conf]/File[/etc/systemd/system/puppetserver.service.d/limits.conf]: The container Systemd::Dropin_file[puppetserver.service-limits.conf] will propagate my refresh event
  Debug: Systemd::Dropin_file[puppetserver.service-limits.conf]: The container Class[Puppet::Server::Puppetserver] will propagate my refresh event
  Debug: Class[Puppet::Server::Puppetserver]: The container Stage[main] will propagate my refresh event
  Debug: Class[Puppet::Server::Puppetserver]: The container Class[Puppet::Server::Config] will propagate my refresh event
  Debug: Class[Puppet::Server::Config]: The container Stage[main] will propagate my refresh event
  Debug: Class[Puppet::Server::Config]: The container Class[Puppet::Server] will propagate my refresh event
  Info: Class[Puppet::Server::Config]: Scheduling refresh of Class[Puppet::Server::Service]
  Info: Class[Puppet::Server::Service]: Scheduling refresh of Service[puppetserver]
  Debug: Executing: '/usr/bin/systemctl is-active -- puppetserver'
  Debug: Executing: '/usr/bin/systemctl is-enabled -- puppetserver'
  Debug: Executing: '/usr/bin/systemctl is-active -- puppetserver'
  Debug: Executing: '/usr/bin/systemctl show --property=NeedDaemonReload -- puppetserver'
  Debug: Executing: '/usr/bin/systemctl restart -- puppetserver'

However, when I run it later:

# systemctl cat puppetserver
# Warning: puppetserver.service changed on disk, the version systemd has loaded is outdated.
# This output shows the current version of the unit's original fragment and drop-in files.
# If fragments or drop-ins were added or removed, they are not properly reflected in this output.
# Run 'systemctl daemon-reload' to reload units.
# /lib/systemd/system/puppetserver.service
...
# systemctl show --property=NeedDaemonReload -- puppetserver
NeedDaemonReload=yes
# cat /etc/systemd/system/puppetserver.service.d/limits.conf
[Service]
LimitNOFILE=32143

This looks like a real edge case and I'm not sure where it comes from. Race condition with systemd itself?

ehelms · 2021-07-26T15:54:16Z

This looks like a real edge case and I'm not sure where it comes from. Race condition with systemd itself?

Fun. I had seen this similar error with a Puppet 5 test so I agree it must be a bit of a race condition.

ehelms

Will let you merge in case anything needs to be squashed.

ekohl · 2021-07-26T16:24:09Z

I found a minimal reproducer:

# ls /etc/systemd/system/puppetserver.service.d
limits.conf
mv /etc/systemd/system/puppetserver.service.d ~ && systemctl show --property=NeedDaemonReload -- puppetserver
NeedDaemonReload=yes
# systemctl daemon-reload
# mv ~/puppetserver.service.d /etc/systemd/system/puppetserver.service.d && systemctl show --property=NeedDaemonReload -- puppetserver
NeedDaemonReload=no
# systemctl cat puppetserver
# /lib/systemd/system/puppetserver.service
#
# Local settings can be configured without being overwritten by package upgrades, for example
# if you want to increase puppetserver open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/puppetserver.service.d/limits.conf" containing:
#       [Service]
#       LimitNOFILE=10000
# You can confirm it worked by running systemctl daemon-reload
# then running systemctl show puppetserver | grep LimitNOFILE
#
[Unit]
Description=puppetserver Service
After=syslog.target network.target nss-lookup.target 

[Service]
Type=forking
EnvironmentFile=/etc/default/puppetserver
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PIDFile=/var/run/puppetlabs/puppetserver/puppetserver.pid

# https://tickets.puppetlabs.com/browse/EZ-129
# Prior to systemd v228, TasksMax was unset by default, and unlimited. Starting in 228 a default of '512'
# was implemented. This is low enough to cause problems for certain applications. In systemd 231, the
# default was changed to be 15% of the default kernel limit. This explicitly sets TasksMax to 4915,
# which should match the default in systemd 231 and later.
# See https://github.com/systemd/systemd/issues/3211#issuecomment-233676333
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027


ExecReload=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver reload
ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start
ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop

KillMode=process

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target
# systemctl daemon-reload
# systemctl cat puppetserver
# /lib/systemd/system/puppetserver.service
#
# Local settings can be configured without being overwritten by package upgrades, for example
# if you want to increase puppetserver open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/puppetserver.service.d/limits.conf" containing:
#       [Service]
#       LimitNOFILE=10000
# You can confirm it worked by running systemctl daemon-reload
# then running systemctl show puppetserver | grep LimitNOFILE
#
[Unit]
Description=puppetserver Service
After=syslog.target network.target nss-lookup.target 

[Service]
Type=forking
EnvironmentFile=/etc/default/puppetserver
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PIDFile=/var/run/puppetlabs/puppetserver/puppetserver.pid

# https://tickets.puppetlabs.com/browse/EZ-129
# Prior to systemd v228, TasksMax was unset by default, and unlimited. Starting in 228 a default of '512'
# was implemented. This is low enough to cause problems for certain applications. In systemd 231, the
# default was changed to be 15% of the default kernel limit. This explicitly sets TasksMax to 4915,
# which should match the default in systemd 231 and later.
# See https://github.com/systemd/systemd/issues/3211#issuecomment-233676333
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027


ExecReload=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver reload
ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start
ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop

KillMode=process

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/puppetserver.service.d/limits.conf
[Service]
LimitNOFILE=32143

So it looks like the NeedDaemonReload value is not correct if you create a directory with a file in it. I suspect a create directory -> create override may be the same problem.

For now I'll add the daemon reload as a workaround, but this may be a systemd bug.

Config is testing the same thing latest and there's no need to clean the entire puppetserver in between. This speeds up the execution time. The upgrade tests are combined into a single file and also includes Puppet 7 upgrade tests now.

It can be determined on the fly. This will be needed if the ca directory becomes configurable.

Soon this will be needed in the ssl file location determination.

This is closer to the real version.

In Puppetserver 7 the CA directory was changed from $ssldir/ca to $puppetserver_dir/ca. This makes it safer to wipe $ssldir without also wiping your entire CA.

theforeman-bot added Needs testing Not yet reviewed labels Mar 8, 2021

ekohl mentioned this pull request Mar 8, 2021

Add Ruby 2.7 testing to installer jobs theforeman/jenkins-jobs#5

Merged

ekohl marked this pull request as draft March 10, 2021 11:04

ekohl force-pushed the puppet-7 branch from 7d42565 to c337972 Compare March 10, 2021 11:06

ekohl mentioned this pull request Mar 24, 2021

(#780) puppetserver: don't purge metrics.conf / Use dedicated parameter for jruby profiler #781

Merged

mattock reviewed Apr 6, 2021

View reviewed changes

manifests/server/puppetserver.pp Outdated Show resolved Hide resolved

wbclark mentioned this pull request May 4, 2021

Drop EL6 support #785

Closed

ekohl force-pushed the puppet-7 branch 5 times, most recently from 5831503 to ab542a4 Compare June 4, 2021 15:10

ehelms reviewed Jul 19, 2021

View reviewed changes

manifests/server/config.pp Outdated Show resolved Hide resolved

ekohl force-pushed the puppet-7 branch from ab542a4 to 68889e0 Compare July 19, 2021 14:55

ehelms mentioned this pull request Jul 19, 2021

Release 16.0.0 #795

Closed

1 task

ekohl commented Jul 19, 2021

View reviewed changes

manifests/server/config.pp Show resolved Hide resolved

ekohl force-pushed the puppet-7 branch from 68889e0 to 372962d Compare July 23, 2021 15:16

ekohl marked this pull request as ready for review July 23, 2021 15:17

ekohl added the Enhancement label Jul 23, 2021

ekohl commented Jul 23, 2021

View reviewed changes

ehelms reviewed Jul 23, 2021

View reviewed changes

ekohl force-pushed the puppet-7 branch from 372962d to 62c252a Compare July 25, 2021 16:08

ekohl mentioned this pull request Jul 26, 2021

Puppet 7 prep #797

Closed

ekohl added 6 commits July 26, 2021 16:15

Do not test Puppetserver 5 on Debian 10

dc8dbac

This was never built so there's nothing to test

Drop old Puppet 3 compat code in tests

04dadd2

Share more variables in server spec

34ea215

Always set root as dir owner

e29abf4

This was already true for AIO packages, but at least in Debian 10 and Fedora 33's non-AIO packages this is also true.

Handle auth.conf in Puppetserver 7

ce42888

In Puppetserver 7 the legacy auth.conf has been removed. This also means the use-legacy-auth-conf setting is useless.

Add missing puppet-ca endpoints in auth.conf

8ef316b

These are present in the default Puppetserver 7 auth.conf.

ekohl force-pushed the puppet-7 branch from 62c252a to 71a09ed Compare July 26, 2021 14:16

ehelms approved these changes Jul 26, 2021

View reviewed changes

theforeman-bot removed the Not yet reviewed label Jul 26, 2021

ekohl added 5 commits July 26, 2021 18:27

Combine acceptance test files

28db556

Config is testing the same thing latest and there's no need to clean the entire puppetserver in between. This speeds up the execution time. The upgrade tests are combined into a single file and also includes Puppet 7 upgrade tests now.

Make server_ssl_chain_filepath undef by default

818bf00

It can be determined on the fly. This will be needed if the ca directory becomes configurable.

Move real_puppetserver_version definition to the top

2ebc227

Soon this will be needed in the ssl file location determination.

Use the real_puppetserver_version variable

572c20d

This is closer to the real version.

Handle the separate cadir in Puppetserver 7

bdbcecd

In Puppetserver 7 the CA directory was changed from $ssldir/ca to $puppetserver_dir/ca. This makes it safer to wipe $ssldir without also wiping your entire CA.

ekohl force-pushed the puppet-7 branch from 71a09ed to ab5fe2a Compare July 26, 2021 16:29

Support Puppet 7

3e35283

ekohl force-pushed the puppet-7 branch from ab5fe2a to 3e35283 Compare July 26, 2021 18:00

ekohl merged commit 3e35283 into theforeman:master Jul 26, 2021

ekohl deleted the puppet-7 branch July 26, 2021 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Puppet 7 #779

Support Puppet 7 #779

ekohl commented Mar 8, 2021 •

edited

Loading

mattock commented Mar 24, 2021

ekohl commented Mar 24, 2021

ekohl commented Mar 24, 2021

ekohl commented Jun 4, 2021

ekohl commented Jul 23, 2021

ekohl Jul 23, 2021

ehelms Jul 23, 2021

ekohl Jul 25, 2021

ekohl commented Jul 25, 2021

ekohl commented Jul 26, 2021

ehelms commented Jul 26, 2021

ekohl commented Jul 26, 2021

ehelms commented Jul 26, 2021

ehelms left a comment

ekohl commented Jul 26, 2021

Support Puppet 7 #779

Support Puppet 7 #779

Conversation

ekohl commented Mar 8, 2021 • edited Loading

mattock commented Mar 24, 2021

ekohl commented Mar 24, 2021

ekohl commented Mar 24, 2021

ekohl commented Jun 4, 2021

ekohl commented Jul 23, 2021

ekohl Jul 23, 2021

Choose a reason for hiding this comment

ehelms Jul 23, 2021

Choose a reason for hiding this comment

ekohl Jul 25, 2021

Choose a reason for hiding this comment

ekohl commented Jul 25, 2021

ekohl commented Jul 26, 2021

ehelms commented Jul 26, 2021

ekohl commented Jul 26, 2021

ehelms commented Jul 26, 2021

ehelms left a comment

Choose a reason for hiding this comment

ekohl commented Jul 26, 2021

ekohl commented Mar 8, 2021 •

edited

Loading