TripleO and Sensu without ssl, ssl still enabled in rabbitmq.json

[davidmnoriega]

Description of problem

• What did you do?
  Use OpenStack TripleO+OpsTools to deploy OpenStack, with monitoring enabled but no SSL
  https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/environments/monitoring-environment.yaml

• What happened?
  Sensu agents are not able to communicate with rabbitmq

• What did you expect to happen?
  Sensu agents to be able to communicate

• How can someone reproduce the problem?
  Deploy OpenStack using TripleO with monitoring but set MonitoringRabbitUseSSL to false(or not set at all as its default value is false)

Command used and debugging output

Sensu agent logs

{"timestamp":"2019-01-15T15:48:01.357967-0800","level":"warn","message":"transport connection error","reason":"tcp connection lost"}

Platform and version information

• Your OS: CentOS 7.4.1708
• Your Ruby version: ruby-2.0.0.648-33.el7_4.x86_64
• Your version of Puppet: puppet-4.8.2-1.el7.noarch
• Your version of Sensu: sensu-0.27.0-1.el7.noarch
• Your version of this module: puppet-sensu-2.30.1-0.20171102090215.c99f0d2.el7.centos.noarch
• Pike version of TripleO/OpenStack

Anything else to add that you think will be helpful?

Removing ssl{} from sensu's rabbitmq config fixed it and the agents are able to communicate with the sensu server. I dont know much about ruby, but my hunch is this bit of code https://github.com/sensu/sensu-puppet/blob/master/lib/puppet/provider/sensu_rabbitmq_config/json.rb#L35-L55
It looks like it only checks the existence of the key ssl in the hiera data, not it's value? So even though we are setting that to false via TripleO, somehow the sensu puppet module is putting ssl{} into the config, enabling ssl. But like I mention, I dont know ruby, so I could be wrong on that.

[ghoneycutt]

Thank you for the issue, we are looking into it.

[treydock]

@davidmnoriega Could you provide me a way to reproduce having ssl key added to rabbitmq.json? One of the test configs we use in Vagrant uses default of false for sensu_rabbitmq_config property of ssl_transport and that is removing ssl{} from the config file.

https://github.com/sensu/sensu-puppet/blob/master/tests/sensu-server.pp#L4-L19

Results in this:

[root@sensu-server ~]# cat /etc/sensu/conf.d/rabbitmq.json
{
  "rabbitmq": {
    "port": 5672,
    "host": "127.0.0.1",
    "user": "sensu",
    "password": "correct-horse-battery-staple",
    "vhost": "/sensu",
    "heartbeat": 30,
    "prefetch": 1
  }
}

If your set sensu::rabbitmq_ssl_private_key or sensu::rabbitmq_ssl_cert_chain then the ssl key is added regardless of sensu::rabbitmq_ssl.

[ghoneycutt]

A link to the puppet configuration that you are using would be very helpful.

[davidmnoriega]

We add this to our TripleO deploy data:

resource_registry:
  OS::TripleO::Services::SensuClient: ../puppet/services/monitoring/sensu-client.yaml

parameter_defaults:
  MonitoringRabbitHost: x.x.x.x
  MonitoringRabbitPort: 5672
  MonitoringRabbitUserName: admin
  MonitoringRabbitPassword: xxxxx
  MonitoringRabbitUseSSL: false
  MonitoringRabbitVhost: "/sensu"
  SensuClientCustomConfig:
    api:
      warning: 10
      critical: 20

TripleO deployment then renders the per node hieradata on each node and the sensu data is in /etc/puppet/hieradata/service_configs.json:

    "sensu::api": false,
    "sensu::client": true,
    "sensu::client_custom": {
        "api": {
            "critical": 20,
            "warning": 10
        },
        "openstack": {
            "auth_url": "http://172.16.235.200:5000/v2.0",
            "password": "xxxxx",
            "region": "regionOne",
            "tenant_name": "admin",
            "username": "admin"
        }
    },
    "sensu::enterprise": false,
    "sensu::enterprise_dashboard": false,
    "sensu::install_repo": false,
    "sensu::manage_user": false,
    "sensu::rabbitmq_host": "x.x.x.x",
    "sensu::rabbitmq_password": "xxxxx",
    "sensu::rabbitmq_port": 5672,
    "sensu::rabbitmq_ssl": false,
    "sensu::rabbitmq_ssl_cert_chain": "",
    "sensu::rabbitmq_ssl_private_key": "",
    "sensu::rabbitmq_user": "admin",
    "sensu::rabbitmq_vhost": "/sensu",
    "sensu::redact": [
        "password",
        "passwd",
        "pass",
        "api_key",
        "api_token",
        "access_key",
        "secret_key",
        "private_key",
        "secret"
    ],
    "sensu::sensu_plugin_name": "rubygem-sensu-plugin",
    "sensu::sensu_plugin_provider": "yum",
    "sensu::server": false,
    "sensu::version": "present",

Yet ssl{} still shows up in /etc/sensu/conf.d/rabbitmq.json:

{
  "rabbitmq": {
    "port": 5672,
    "host": "x.x.x.x",
    "user": "admin",
    "password": "xxxxx",
    "vhost": "/sensu",
    "heartbeat": 30,
    "prefetch": 1,
    "ssl": {
    }
  }
}

[ghoneycutt]

Could you please upgrade the module to the latest release and see if this is still an issue.

[davidmnoriega]

That might be tricky because the version of openstack determines which CentOS repo is used, which is what determines package versions of things like the puppet-sensu package. I'll see if I can just install a newer version, but that might not be easy to do.

Re-reading @treydock 's comment, I have a hunch that because the TripleO heat templates are building hieradata that set rabbitmq_ssl_cert_chain and rabbitmq_ssl_private_key to "", and because I've just learned that in Ruby, an empty string is evaluated as truthy 😑 , that might be why ssl{} ends up in rabbitmq.json. I modified the heat template and ran the deploy last night and now rabbitmq.json does not turn on ssl, and the agent log shows it is able to communicate with the opstools monitoring server

[treydock]

The issue is actually here: https://github.com/sensu/sensu-puppet/blob/master/manifests/rabbitmq/config.pp#L15. In Puppet empty strings will evaluate to true and once that's done the value for ssl is enabled here: https://github.com/sensu/sensu-puppet/blob/master/manifests/rabbitmq/config.pp#L94

The default for the underlying sensu_rabbitmq_config type is actually empty strings which do not auto-enable SSL. The issue is actually the sensu::rabbitmq::config class which is enabling SSL even when chain and key are empty strings.

[davidmnoriega]

I did try using a newer version of the package, puppet-sensu-2.53.0-1.338ae0egit.el7, and while I could manually install it, it has dependencies on a newer version of puppet than what the pike repos have available, so the puppet apply failed.

I'll be submitting a patch to TripleO with my workaround(dont put cert/key into hiera if ssl is false) as I dont know how an updated sensu-puppet package would get distributed and backported into the various repos.

Thanks

[treydock]

@davidmnoriega There is a patch in #1053 that would require no changes to TripleO. It would prevent this module from enabling SSL when both cert and key are empty strings.