Run a current version of Ansible in the Admin Workstation

[conorsch]

At time of writing, Tails 1.6 ships with Ansible 1.7.2 available in the Debian stable repository. Feature-wise, 1.7.2 is recent enough for our playbooks—the apache2_module was added in 1.6, for example—but it lacks several important bugfixes.

One example is Get MemoryError when fetching large file, which was patched as of 1.7.3. I've seen this bug cause failures with the automatic backup role that runs at the end of the playbook. The failure aborts the backup process, and no encrypted tarball is written to the Admin Workstation. In order to force the backup, a manual rsync command is required, which sidesteps the MemoryError exception.

The problem is how to install a recent version of Ansible safely under Debian wheezy (which Tails 1.6 is based on). The Ansible install docs include a long list of installation methods, but none includes GPG verification, which the current apt-get install process provides.

[garrettr]

Potential solution worth investigating: as mentioned in #1035 (comment), Tails includes the Debian testing repos in their apt sources list by default. The current version of Ansible in Debian testing is 1.9.4.

[conorsch]

Yes, sudo apt-get install -y -t testing ansible works! Haven't tested apt persistence in Tails with more recent version, will try that next.

[garrettr]

@conorsch I bet the apt persistence would work just fine for packages installed from the testing repos. Have you had a chance to test it yet?

[garrettr]

You should also test the workflow for migrating Admin Workstations that were using ansible from the stable repos, since that package will be cached locally due to the apt persistence. Can they just run apt-get install -t testing ansible, or do they need to explicitly remove the stable ansible package first?

[conorsch]

@garrettr Haven't tested sufficiently yet. Persistence seems to work fine, but as you mention, it may be the stable version getting installed—haven't confirmed that it's the 1.9.x release that's getting reinstalled.

Also, installing ansible from the testing repo results in several interactive dpkg prompts. Sticking to the default suggestions for the confirmation prompts works fine, but will those prompts cause problems with apt persistence? Hopefully the Tails apt persistence correctly uses a Debian noninteractive environment, but will need to confirm to be sure.

[psivesely]

I installed testing/ansible on our test instance Admin Workstation. After reboot it reinstalled oldstable/ansible.

[heartsucker]

Commenting here since I just got bit by this while testing something. If we update, can we at least push over to something from the 2.1.x line? This is mostly backwards compatible with 1.8.4 (current recommendation), and will at at least get us the new goodies and syntax. This can be installed from pip easily (but not apt) for Debian 8 based systems.

[conorsch]

Need to loop back on this. We're still expecting a very old version of Ansible, and if persistence works with a 2.x release of Ansible, that's ideal. If not, I'd be willing to fall back to a pip install with an explicit version requirement. We'd lose GPG verification via apt, but with Tails forcing all connections over Tor, and pip now supporting TLS connections, we can reasonably assume it would be prohibitively difficult to perform a targeted MITM attack on the ansible install.

[psivesely]

Does Tails come w/ python-pip installed? If not is the plan to install that first and then use it to install Ansible? In Debian Jessie python-pip is currently version 1.5.6. I think that verison use HTTPS by default for PyPi, but it wouldn't hurt to confirm.

[psivesely]

There's an official Ansible PPA that would allow us to maintain GPG verification, while providing the latest stable Ansible release.

[conorsch]

There's an official Ansible PPA that would allow us to maintain GPG verification, while providing the latest stable Ansible release.

@fowlslegs I like that idea, mainly because it's technically more secure than pip installations, but we'd still need to bootstrap the configuration of that repo without Ansible to get it installed and configured. Given rickety state of the install.sh bash script, those changes should be very well tested.

[psivesely]

We talked and decided that with the precarious state of Tails APT sources and preferences lists, we are weary of further complicating this.

[conorsch]

Relevant discussion ongoing in #1435.

[conorsch]

First-time Admins frequently install Ansible v2.x via pip and then report problems, so updating the playbooks and just using the new-style syntax seems the best way to go. We can provide a fail task during validation to bail out of an old version of Ansible is used, so we don't have to support syntaxes for both versions.

[psivesely]

^ Seems like the way to go. Do you think we should persistently install Ansible on the AW, or just have admins install it ad-hoc when they need to run playbooks?

[conorsch]

We'd have to:

1. Ensure Ansible via apt is not installed
2. Ensure the ansible package is absent from apt persistence settings
3. Install python-pip via apt
4. Install Ansible via python-pip

Theoretically that's all possible in a run of the ./install.sh script, but it will naturally require explicit docs. This has me thinking more about the underlying problem, which is our shoehorning persistent system configs into an amnesiac OS....

[ageis]

@conorsch @redshiftzero Hey guys, is this rsync issue supposed to be fixed in develop branch with the trusty bento box and Ansible 2.3.1 / Vagrant 1.9.7? I haven't been able to provision staging...

TASK [build-securedrop-app-code-deb-pkg : Copy install_files/securedrop-app-code dir to build path.] ***
fatal: [build]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --delete-after --archive --rsh=/usr/bin/ssh -S none -i /home/kevin/.vagrant.d/insecure_private_key -o Port=2200 -o StrictHostKeyChecking=no --rsync-path='sudo rsync' --out-format=<<CHANGED>>%i %n%L /home/kevin/dev/securedrop/install_files/securedrop-app-code/ [email protected]:/tmp/securedrop-app-code-0.3.12-amd64", "failed": true, "msg": "bash: sudo rsync: command not found\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: remote command not found (code 127) at io.c(235) [sender=3.1.2]\n", "rc": 127}

[redshiftzero]

What version of rsync do you have? If it's 2.x you might need to install a more recent version. I'm using version 3.1.2, which I installed using brew.

[conorsch]

If @ageis can confirm it's an outdated rsync version causing the problem, we should probably add a quick check to the build playbook that will fail with an informative error message. Also, note that the dev docs stipulate running pip install -r securedrop/requirements/develop-requirements.txt, which pegs Ansible at 2.2.1—which may change, but it's the "correct" version for current development.

[ageis]

I don't think that's the problem, I figured it was that privilege issue mentioned in the related commit; I'm running rsync 3.1.2 which is the absolute latest on my host machine. The VM has 3.1.0. It would seem to be because of something that's changed in Ansible, because 2.2.1 works for me. Thought I had tried downgrading to that before but I guess not.

It might be this bug which has been filed against Ansible 2.3.x: ansible/ansible#23575 Hopefully the fix will land in an upcoming point release.

Thanks!