gems not installed on sensu upgrade.

[fessyfoo]

Description of problem

I'm wondering if anyone else has run into this problem. On a recent rather big jump upgrade of sensu 0.26.0.1 to 1.4.2.1 on our windows clients. On the first puppet run sensu is upgraded and the embedded ruby is upgraded and until the next puppet run our gem plugins are no longer installed.

Command used and debugging output

We're basically using the sensu class to manage sensu.
and we use the sensu_gem provider on package resources to install sensu client plugins and they have an ordering constraint on Package[sensu].
for example:

  package { 'sensu-plugins-windows':
    ensure   => '0.0.8',
    provider => 'sensu_gem',
    require  => Package['sensu'],
  }

example puppet run info:

Notice: /Stage[main]/Sensu::Package/Package[sensu]/ensure: ensure changed '0.26.0.1' to '1.4.2.1'
Notice: /Stage[main]/Sensu::Client::Service/Service[sensu-client]: Triggered 'refresh' from 1 events

after which the plugin we were using sensu-plugins-windows is no longer installed. and sensu checks start failing (UNKNOWN) which across a large number of machines is a lot of alerting noise.

the next puppet run will reinstall it.

Notice: /Stage[main]/Monitoring_profiles::Plugins::Windows_checks/Package[sensu-plugins-windows]/ensure: created

Initially we assumed we had some puppet ordering constraint incorrect, but another theory is that the puppet base Package provider does prefetching of resources to initialize providers for all existent resources on a box. This is apparently done at the first occurrence of of a Package resource. (which I either found spelunking puppet code or in this wacky article ) and because this state is cached it's from before the embeded sensu upgrade, and therefore the current puppet run can't realize that the sensu plugin is not in the new version of ruby's gems directories.

Now if this were 100% correct then we should have had the same problems on our linux nodes. Which we did not. But that doesn't 100% disprove the theory either could be another reason the linux nodes worked.

From as best as I could understand the prefetch code in puppet it does seem like this is an issue.

Anyone else experience anything like this ever?

Thoughts on how to properly quantify the issue? How to properly fix it?

Platform and version information

• Your OS: win2k8, win2k12 [ also ubuntu linux but they didn't fail. ]
• Your Ruby version: embedded with sensu.
• Your version of Puppet: 3.7.1
• Your version of Sensu: 0.26.0 and 1.4.2
• Your version of this module: opentable:f407caedb6743d81710fe46ce7b7496616a90a19(slight fork, hoping that's not involved, also old, if that's involved then yay upgrade! . )

[fessyfoo]

cc @SegFaultAX @nadarashwin @travees @kstdennis @dcharleston @carlf @vlau-opentable @jay-chin @vishalud

[ghoneycutt]

Hi @fessyfoo Please try using our latest release of the module and let us know if this is still an issue. We do not have the resources to troubleshoot your fork of the software.

[fessyfoo]

right, i didn't notice we were using a forked version until the end of writing this ticket. we'll need to work that out and get back on upstream.

still interested if anyone has noticed this type of problem because it looks to be more an issue with the upstream puppet gem provider code. if you change out ruby in the middle of a puppet run and you have cached the state of gems before that. you're forced into a two run puppet situation.

[ghoneycutt]

Closing this ticket since it is not verified with the latest version of the module. We value your input and appreciate you opening tickets with us. Please re-open if this is still an issue.