Puppet 6.2 fails to apply sensu config, failing with: "Error 500 on SERVER: Server Error: no parameter named 'custom' (file: /etc/puppetlabs/code/environments/ta _dit_production/modules/sensu/manifests/enterprise/dashboard.pp, line: 56) on Sensu_enterprise_dashboard_config"

[jwoytek]

Description of problem

• What did you do?
  Tried to apply existing sensu config after an update to Puppet 6.2
• What happened?
  Update did not apply, failing with:
  Error 500 on SERVER: Server Error: no parameter named 'custom' (file: /etc/puppetlabs/code/environments/ta
  _dit_production/modules/sensu/manifests/enterprise/dashboard.pp, line: 56) on Sensu_enterprise_dashboard_config`
• What did you expect to happen?
  Update should apply cleanly.
• How can someone reproduce the problem?
  Attempt to apply a sensu config with Puppet 6.2 and sensu-puppet 2.62.3.

Command used and debugging output

• What does your puppet config look like (including any hiera config)
  [not able to share]
• Is this a masterless or master based puppet setup?
  Master

Platform and version information

• Your OS:
  RHEL 7.5
• Your Ruby version:
• Your version of Puppet:
  Open Source 6.2
• Your version of Sensu:
  0.26.5-2
• Your version of this module:
  2.62.3

Anything else to add that you think will be helpful?

I patched around what I think is the issue. Pull request to come.

[treydock]

@jwoytek What does your Puppet code look like that is invoking sensu_enterprise_dashobard_config or sensu::enterprise::dashboard or just the main sensu class?

[treydock]

I just ran acceptance tests locally with Puppet 6.2.0 and defined this:

         class { 'sensu':
            enterprise           => true,
            enterprise_dashboard => true,
            enterprise_dashboard_custom => {
              'usersOptions' => { 'requireSilencingReason' => true },
            },
            enterprise_user      => '#{ENV['SE_USER']}',
            enterprise_pass      => '#{ENV['SE_PASS']}',
            rabbitmq_password    => 'secret',
            rabbitmq_host        => '127.0.0.1',
          }
          sensu::enterprise::dashboard::api { 'sensu.example.com':
            datacenter => 'example-dc',
          }

I got no errors. The error is like the type code isn't in sync with module because manifest is calling custom property and error says doesn't exist in type.

[jwoytek]

We're not even invoking enterprise or enterprise_dashboard. Those values are unconfigured in our sensu class definition, which itself is pretty vanilla. We define the rabbitmq info and a couple of other bits. I can't access it right this second but will see if I can grab and redact it in a little while.

I did not understand the intent of the different way custom was handled from everything else in the ruby code, so I understand that the simplistic fix is probably not ideal.

[jwoytek]

@treydock we configure the sensu module through hiera. Our client configuration is pretty simple:

sensu::server: false
sensu::install_repo: false
sensu::purge:
  config: true
sensu::rabbitmq_host: [redacted]
sensu::rabbitmq_password: [redacted]
sensu::rabbitmq_port: 5672
sensu::rabbitmq_vhost: /sensu

[treydock]

What's the server configuration look like? That bit of Hiera wouldn't invoke any enterprise features.

[jwoytek]

@treydock FTR, on the hosts where we were seeing these errors, the sensu server configuration is not applied. It, too, however, is pretty simple:

sensu::install_repo: false
sensu::purge:
  config: true
sensu::rabbitmq_host: [redacted]
sensu::rabbitmq_password: [redacted]
sensu::rabbitmq_port: 5672
sensu::rabbitmq_vhost: /sensu
sensu::server: true
sensu::redis_host: [redacted]
sensu::api: false

[jwoytek]

@treydock additionally, we have a separate API server, which gets this config applied:

sensu::install_repo: false
sensu::purge:
  config: true
sensu::redis_host: [redacted]
sensu::api: true

[treydock]

I think because you set sensu::purge the code in sensu::enterprise::dashboard is getting executed with ensure => 'absent' which explains why that code path is taken. Did you update the sensu puppet module recently? The change to add custom to sensu_enterprise_dashboard_config was done in version v2.59.0. I am wondering if your hitting issue where new type properties aren't picked up by puppetserver or if your using environment caching maybe the cache is out of date if you recently updated the module.

[jwoytek]

@treydock the group that maintains our puppet server just recently updated to 6.2 (from 5), and introduced some problems when they did. When they were fixing those problems, they did update the sensu module from 2.58.0 to 2.62.3.

Interestingly, the failure was inconsistent: Sometimes the configuration would apply, and other times not, and I was unable to tie success to any specific variable. However, your idea on sensu:purge makes sense, and would help to explain why we were taking the code path for something we are not using.

Is there a better way to do the check as originally intended in the code that would fail gracefully in that condition like the other members of that config object?

[treydock]

The error your seeing is more than likely something on the puppetserver side, I honestly don't think there's anything we could do on sensu module to stop that error since the error is because the master doesn't see that sensu_enterprise_dashboard_config has the custom property. The fact you recently updated the module from a version without that property one with the property is a big clue. I can't explain the inconsistent behavior unless your using multiple masters and one is out of sync.

I'd check to ensure puppetserver hosts have their plugins synced (just run puppet on them or run plugin sync) and also may help to restart puppetserver after it's verified that the types are synced. Also check if the puppetserver is using puppet generate types and that the type cache is up-to-date.

Some other debugging would be inspecting the lib/puppet/type/sensu_enterprise_dashboard_config.rb file on puppet server to ensure the custom property is defined in the file. The file would be in two places, first would be modulepath (like /etc/puppetlabs/code) and the other would be location used by plugin sync in /opt/puppetlabs/puppet.

[jwoytek]

@treydock OK, well, we don't have direct access to the puppet server, but I'll see if we can get the group that manages it to restart it. It is a single server, no multiple masters.

I don't know if this tells you anything interesting, but our local cache on the client side while we were having the problems did have the file sensu_enterprise_dashboard_config.rb, and it did define custom.

Now that all of our systems have updated with my patched module, though, switching back to the 2.62.3 code seems to present no problems, and configurations apply correctly.

Now that I can't reproduce this error, this might be entirely moot. I need to bail for the moment, but I'll be back at this Monday and will run through some less well-used systems to see if I can trigger the problem again so I can get the code out of the cache to see if it differs from the code in cache on a working system. If I can't duplicate the issue, I'll update here and will close this issue and withdraw the associated PR.

[jwoytek]

@treydock apologies for not updating this sooner. I meant to do that and forgot to come back. The manager of the upstream puppet master reset the server and made sure that the type cache had been updated. After doing that and swapping back to the 2.62.3 code without my patch, it looked like everything was working correctly.

We are still unable to explain how the cache apparently wedged when the module was upgraded, but a few service restarts on the puppet server apparently made things happy again.

Thanks for the help tracking down this issue. Hopefully this ticket will help others who might end up in the same situation. Closing this as an active issue.