use of new 6.0.0 module on new installation fails

[bostrowski13]

PE 2017.2
Target OS: RHEL 7.4
RabbitMQ Version: 3.6.3
Erlang Version: 18.3
Module version: puppet-rabbitmq (v6.0.0-rc0)

The first error seen is...
Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method []' for nil:NilClass`

but I believe this is because RabbitMQ is not running, and can't execute rabbitmqctl.

Upon further investigation, looking in the startup_err log, i see...

[root@server1 user1]# cat /var/log/rabbitmq/startup_err
/usr/lib/rabbitmq/bin/rabbitmq-env: line 76: /etc/rabbitmq/rabbitmq-env.conf: Permission denied

I validated that this file exists, and looking at the permissions on /etc/rabbitmq/rabbitmq-env.conf, it

[root@server1 user1]# ll /etc/rabbitmq/
total 20
-rw-r--r--. 1 root root       23 Sep  6 12:29 enabled_plugins
-rw-r-----. 1 root rabbitmq   73 Sep  6 13:28 inetrc
-rw-r-----. 1 root rabbitmq   23 Sep  6 12:29 rabbitmqadmin.conf
-rw-r-----. 1 root rabbitmq 1462 Sep  6 13:28 rabbitmq.config
-rw-r-----. 1 root rabbitmq   39 Sep  6 12:29 rabbitmq-env.conf
drwxr-xr-x. 2 root root       58 Sep  6 12:29 ssl

That file contains...

[root@server1 rabbitmq]# cat rabbitmq-env.conf
export ERL_INETRC=/etc/rabbitmq/inetrc

and the inetrc file exists, as well, and contains nothing (except the puppet header)...

[root@servver1 rabbitmq]# cat inetrc
% This file managed by Puppet
% Template Path: rabbitmq/templates/inetrc

[bostrowski13]

Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method `[]' for nil:NilClass
Info: Caching catalog for server1.domain.com
Info: Applying configuration version 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Info: Class[Rabbitmq::Service]: Scheduling refresh of Service[rabbitmq-server]
Error: Systemd start for rabbitmq-server failed!

Error: /Stage[main]/Rabbitmq::Service/Service[rabbitmq-server]/ensure: change from stopped to running failed: Systemd start for rabbitmq-server failed!
journalctl log for rabbitmq-server:
-- Logs begin at Wed 2017-09-06 12:23:08 EDT, end at Wed 2017-09-06 13:28:32 EDT. --
Sep 06 13:28:32 server1.domain.com systemd[1]: Starting LSB: Enable AMQP service provided by RabbitMQ broker...
Sep 06 13:28:32 server1.domain.com su[12908]: (to rabbitmq) root on none
Sep 06 13:28:32 server1.domain.com su[12941]: (to rabbitmq) root on none
Sep 06 13:28:32 server1.domain.com su[12945]: (to rabbitmq) root on none
Sep 06 13:28:32 server1.domain.com rabbitmq-server[12833]: Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}
Sep 06 13:28:32 server1.domain.com rabbitmq-server[12833]: rabbitmq-server.
Sep 06 13:28:32 server1.domain.com systemd[1]: rabbitmq-server.service: control process exited, code=exited status=1
Sep 06 13:28:32 server1.domain.com systemd[1]: Failed to start LSB: Enable AMQP service provided by RabbitMQ broker.
Sep 06 13:28:32 server1.domain.com systemd[1]: Unit rabbitmq-server.service entered failed state.
Sep 06 13:28:32 server1.domain.com systemd[1]: rabbitmq-server.service failed.

[bostrowski13]

    class { '::rabbitmq':
        package_ensure           => $rabbitmq_version,
        repos_ensure             => true,
        service_manage           => true,
        delete_guest_user        => true,
        admin_enable             => true,
        config_cluster           => true,
        cluster_nodes            => $cluster_nodes,
        cluster_node_type        => 'disc',
        erlang_cookie            => $erlang_cookie,
        wipe_db_on_cookie_change => true,
        auth_backends            => [rabbit_auth_backend_internal],
        tcp_backlog              => 512,
        default_user             => $default_username,
        default_pass             => $default_password,
        config_variables         => {
            'log_levels'               => '[{connection,info},{channel,info}]',
            'vm_memory_high_watermark' => 0.75,
            'handshake_timeout'        => 30000,
            'ssl_handshake_timeout'    => 10000,
        },
        ssl                      => true,
        ssl_only                 => true,
        ssl_port                 => '5671',
        ssl_versions             => ['tlsv1.2', 'tlsv1.1'],
        ssl_cacert               => '/etc/rabbitmq/ssl/chain.pem',
        ssl_cert                 => '/etc/rabbitmq/ssl/envcert.pem',
        ssl_key                  => '/etc/rabbitmq/ssl/envcert.key',
        ssl_interface            => $::ipaddress,
        ssl_fail_if_no_peer_cert => false,
        management_ssl           => false,
        ssl_depth                => 2,
        ssl_cert_password        => $ssl_cert_password,
        require                  => Package['erlang','socat']
    }

[wyardley]

@bostrowski13 this is all very helpful, will try to reproduce.

Do /var/log/rabbitmq/startup_log / /var/log/rabbitmq/startup_err have anything useful?
Can you post the output of yum info rabbitmq-server?

I think inetrc shoudl be empty unless ipv6 is enabled, so that seems normal.

I think the chicken / egg problems with the facts are also a known issue (since RabbitMQ isn't running, they won't work, including the version one, though we might want to make it a little more tolerant of the failure). Are you setting $rabbitmq_version elsewhere, though (from a site manifest)? I don't think it's valid to use the $rabbitmq_version fact for package_ensure.

[bostrowski13]

]# yum info rabbitmq-server
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
Repository 'rabbitmq_rabbitmq-server' is missing name in configuration, using id
Installed Packages
Name        : rabbitmq-server
Arch        : noarch
Version     : 3.6.3
Release     : 1
Size        : 5.9 M
Repo        : installed
From repo   : rabbitmq_rabbitmq-server
Summary     : The RabbitMQ server
URL         : http://www.rabbitmq.com/
License     : MPLv1.1 and MIT and ASL 2.0 and BSD
Description : RabbitMQ is an open source multi-protocol messaging broker.

Available Packages
Name        : rabbitmq-server
Arch        : noarch
Version     : 3.6.11
Release     : 1.el7
Size        : 4.7 M
Repo        : rabbitmq_rabbitmq-server/x86_64
Summary     : The RabbitMQ server
URL         : http://www.rabbitmq.com/
License     : MPLv1.1 and MIT and ASL 2.0 and BSD
Description : RabbitMQ is an open source multi-protocol messaging broker.

startup_log contains notning, err contains the line i mentioned above.

we do have ipv6 disabled, and I saw this should be empty, just trying to be thorough.

No, that is where we are setting the version number to pin 3.6.3. I thought i recently asked or read something about the depricated used of "$version", and to pin it was to use this param instead of that.

[bostrowski13]

pin the version, without using "version" and using "package_ensure" was in #154

[wyardley]

Correct, yeah, I just wanted to make sure you weren't using the fact from the module to pin, since this would create a chicken / egg situation. I'm not sure if the $rabbitmq_version variable will conflict with the rabbitmq_version fact, hopefully won't be an issue.

In addition, depending on the package source, you may need a version suffix (the yum provider is pretty picky about versions matching the package name; sometimes it needs to have the release tag added).

Is rabbitmq user in the rabbitmq group (groups rabbitmq)? If you switch to the rabbitmq user (su -s/bin/bash - rabbitmq, can you cat that file? I think the main issue seems to be permissions on that file. With the default setup, it should be able to read the file, I think.

[wyardley]

fwiw, on CentOS 7:

[root@centos-7-x64 tmp]# cat /tmp/apply_test_611 
  class { '::rabbitmq':
        package_ensure           => '3.6.3',
        repos_ensure             => true,
        service_manage           => true,
        delete_guest_user        => true,
        admin_enable             => true,
        config_cluster           => true,
        cluster_nodes            => [],
        cluster_node_type        => 'disc',
        erlang_cookie            => 'DAF5938743',
        wipe_db_on_cookie_change => true,
        auth_backends            => [rabbit_auth_backend_internal],
        tcp_backlog              => 512,
        default_user             => 'rabbitmq',
        default_pass             => 'asdf123',
        config_variables         => {
            'log_levels'               => '[{connection,info},{channel,info}]',
            'vm_memory_high_watermark' => 0.75,
            'handshake_timeout'        => 30000,
            'ssl_handshake_timeout'    => 10000,
        },
        ssl                      => true,
        ssl_only                 => true,
        ssl_port                 => '5671',
        ssl_versions             => ['tlsv1.2', 'tlsv1.1'],
        ssl_cert                 => '/etc/pki/tls/certs/test.crt',
        ssl_key                  => '/etc/pki/tls/private/test.key',
        ssl_fail_if_no_peer_cert => false,
        management_ssl           => false,
        ssl_depth                => 2,
    }
[root@centos-7-x64 tmp]# userdel rabbitmq ; groupdel rabbimq ; rpm -e rabbitmq-server
userdel: user rabbitmq is currently used by process 24321
groupdel: group 'rabbimq' does not exist
Stopping rabbitmq-server (via systemctl):                  [  OK  ]
[root@centos-7-x64 tmp]# puppet apply apply_test_611 
Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (in /etc/puppetlabs/puppet/hiera.yaml)
Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppetlabs/code/modules/rabbitmq/manifests/init.pp", 99]:
   (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Notice: Compiled catalog for centos-7-x64.ad.axs.com in environment production in 0.67 seconds
Notice: /Stage[main]/Rabbitmq::Install/Package[rabbitmq-server]/ensure: created
Notice: /Stage[main]/Rabbitmq::Service/Service[rabbitmq-server]/ensure: ensure changed 'stopped' to 'running'
Notice: Applied catalog in 13.78 seconds
[root@centos-7-x64 tmp]# su - rabbitmq
Last login: Wed Sep  6 08:06:28 UTC 2017
-bash-4.2$ cat /etc/rabbitmq/rabbitmq-env.conf 
export ERL_INETRC=/etc/rabbitmq/inetrc

It could potentially be directory permissions on /etc/rabbitmq, can you check those too?

[bostrowski13]

so here's a really fun one...

apparently the image of the server once had the pivotal version of rabbitmq installed with the RPM on it, and so the result of the groups command showed...

# groups rabbitmq
rabbitmq : pivotal

after i added the rabbitmq user to the rabbitmq group, no issues at all.

[wyardley]

👍
Awesome
Yeah, the RPM will create the user if it doesn't exist; the module itself doesn't try to manage the RabbitMQ user. Glad we got that figured out.

[bostrowski13]

thanks for your guidance. gotta get a cleaner image from our build team.

[alexjfisher]

@bostrowski13 I don't think the failure in the fact is actually causing you any problems (it's not used). But I would be interested to know why it's throwing an error.
What is the output of rabbitmqctl status? Maybe this regex needs tweaking.

[wyardley]

@bostrowski13 @alexjfisher I think #615 may fix this, in the case where rabbitmqctl is not installed or not installed yet. It seems that the fact should work correctly even when it is not running (though version will not).