does not start at boot on CentOS 6

[petere]

On CentOS 6, this cookbook registers consul as an upstart job. The consul process is started when Chef runs. I can also manually start and stop consul using something like sudo start consul. But after rebooting, consul is not automatically started. I can start it manually again, and everything is fine, but that's probably not the idea.

The configuration in /etc/init/consul.conf seems to indicate that it should automatically start at system start. But the upstart version CentOS 6 is so old, it's hard to debug anything.

Here is a small Vagrantfile to test this:

Vagrant.configure('2') do |config|
  config.vm.box = 'chef/centos-6.6'
  config.vm.provision 'chef_solo' do |chef|
    chef.add_recipe 'consul-cluster'
    chef.json = {
      'consul' => {
        'config' => {
          'enable_syslog' => true
        }
      },
      'consul-cluster' => {
        'config' => {
          'bootstrap_expect' => 1
        }
      },
    }
  end
end

After vagrant up, check that consul is running, then run vagrant reload, and it won't come back up.

[johnbellone]

What's the minor version of CentOS 6 that you're using?

[petere]

6.6, as in the Vagrantfile shown above.

[johnbellone]

I was able to confirm that Consul isn't starting on reboot.

[legal90]

@petere FYI: it could be worked around by switching service provider to sysvinit (instead of upstart):

node.set['poise-service']['consul']['provider'] = 'sysvinit'

Or the same with poise_service_options resource:

poise_service_options node['consul']['service_name'] do
  _options['provider'] = 'sysvinit'
end

[johnbellone]

That was what I was going to suggest as well. I'm not comfortable using the version of upstart on CentOS 6 because it is ancient. I'll try and take some time to mess around with this, but this is probably the best route.

[petere]

I had tried that as well, but it doesn't work. The first problem is that the generated init script does not pass the path (/usr/local/bin) for consul, so it doesn't start. And if you fix that (manually), consul starts, but the init script fails because it checks for the PID of /usr/bin/env. And you don't get any error messages about either of these. It become clear very quickly that that scenario is not tested very well in poise-service. You can easily reproduce this by adding the appropriate attributes to the default test in .kitchen.yml.

[johnbellone]

Alright, we can fix this by putting a custom sysvinit template into the project.

[griffint61]

I saw this too, and managed to figure out what's going on. It's failing because with the upstart config on EL6, Consul is being started before the network is up, and then Consul itself bails out because, again, the network is not up. I would be nice if Consul was a little more patient in this scenario, but there's probably a way to build some patience in to the upstart config, or give it an explicit dependency on networking. Note that networking on EL6 is managed by a sysvinit script.

My workaround for now is have my wrapper cookbook add a :reboot cron job that watches for /var/lock/subsys/network, and then run "initctl start consul" if Consul is not already running.

[johnbellone]

@griffint61 That's interesting. I have a new sysvinit template that I wrote for the project, but if I configure the service to start on run levels 3 and 5 it should be fine?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.