bootstrapping new role[web] nodes fails

[irabinovitch]

I was working on spinning up a new web server today to help the web team test some new configurations and ran into some issues. Some of this may just be docs needing to be updated. Heres a running list of issues I encountered:

Issues

• Step 1: clone scale-chef onto host in /var/chef/repo. This works as expected, but assumes you have git on the box which isn't true until Step 2. We should document this first step or change the docs to copying bootstrap script to the host rather than entire repo, as that script will install git and do the cloning. We can probably even put this into a user-data script and have the machine do it automatically on provisioning.
• Step 2: run chef_boostrap.sh. This fails silently (at least on centos7). The omnibus.cinc.sh cert fails to validate and because we use wget -q here no error is printed out and the script exists as if everything worked fine.
• Step 3: run chefctl -i. Even after a successful bootstrap chefctl isn't present at /usr/local/sbin/chefctl or anywhere else in the path. It looks like you have to run it from /var/chef/repo/cookbooks for the first time.
• Step 4: Once chefctl has been run role[web] fails on the following issues: epel-release-7-12.noarch.rpm cant be found. this should be fixed by update to epel-release-7-14.noarch.rpm #281
• Step 5: a bunch of scripts fail until you sync scale-secrets on the host, but we dont mention that in docs.
• Step 6: scale_apache cookbook fails to run until letsencrypt runs and populates certs on the system. Is there a step missing from docs here?
• Step 7: scale_apache cookbook fails unless you first clone scale-drupal to /home/drupal/. We can probably automate this like we did for scale-legacy-web.
• Step 8: restore static files from backup using restore-drupal-static.py. (this might go away if we finish scale-drupal#102

[irabinovitch]

On Centos8 the cert issues go away during bootstrap, but we probably still want better error handling in the bootstrap script so that its clear we failed.