Refactor to support Ansible 2.8 #1549

jackivanov · 2019-08-15T13:51:46Z

Upgrades and Fixes

Upgrade to Ansible 2.8.3
Upgrade to Python 3
Big documentation changes

Cloud provider changes

Add a new cloud provider - Hetzner Cloud. (Fixes Add Hetzner Cloud #988)
Fix Scaleway instance creating issue. (Fixes Instance creation issue(Scaleway) #1502)
Remove deprecated DigitalOcean (Fixes Digital Ocean Droplet creation failed with DoError: found unpermitted parameters: virtio #1571) and GCE modules

Breaking changes

python virtual environment location is now .env
GCE: from now, the installer expects you to specify a region (eg: us-east1), instead of a zone (eg: us-east1-a).

Types of changes

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

I have read the CONTRIBUTING document.
My code follows the code style of this project.
I have updated the documentation accordingly.
All new and existing tests passed.

davidemyers · 2019-08-15T22:57:10Z

I tried a few tests. Starting with a new Ubuntu Server 19.04 container I ran cloud deployments to Vultr and DigitalOcean. Then I created an Ubuntu Server 19.04 droplet on DigitalOcean and did a local install. All tested fine using WireGuard on iOS.

phaer

Besides adding the api-token to the 2 places described, I had to add a tasks in roles/common/tasks/ubuntu.yaml to wait for unattended-upgrades to finish, which seemed to start as soon as the hcloud server had been provisioned:

    - name: Wait for any possibly running unattended upgrade to finish
      raw: systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true
 
    - name: Install software updates

After that it worked fine.

roles/cloud-hetzner/tasks/main.yml

jackivanov · 2019-08-16T08:46:33Z

Thanks @phaer! Somehow I missed the api tokens. I just sent another solution for the apt lock, works well.

phaer · 2019-08-16T08:52:37Z

I just sent another solution for the apt lock, works well.
Ah yes, thanks! My ansible skills are a bit rusty, didn't think about that.

davidemyers · 2019-08-16T12:06:35Z

An Ubuntu Server 18.04 droplet local install works OK too.

davidemyers · 2019-08-16T12:43:22Z

Also, just for "fun", an Ubuntu Server 19.04 droplet local install works if you replace python with python3 in the setup instructions. 😄

TC1977 · 2019-08-16T15:09:26Z

Update docs?

jackivanov · 2019-08-16T15:12:09Z

@TC1977 I would appreciate your help :) I guess, you can commit directly to this branch?

TC1977 · 2019-08-16T15:33:15Z

Working on it now 😄 I can't commit directly, but I can submit a PR to this branch.

jackivanov · 2019-08-16T15:34:09Z

Yes, that's OK

TC1977 · 2019-08-16T17:34:29Z

Ok, I think I'm all set. #1552

davidemyers · 2019-08-17T19:43:32Z

A cloud deployment from an Ubuntu Server 19.04 container to DigitalOcean using python3 works with these new DO modules.

It works if LANG=C.UTF-8 but fails if LANG="en_US.UTF-8" on this bit where the p12 password is generated:

openssl rand 8 |
      python -c 'import sys,string; chars=string.ascii_letters + string.digits + "_@"; print("".join([chars[ord(c) % 64] for c in list(sys.stdin.read())]))'

Edited to add: On Ubuntu Server 18.04 I needed to use LANG=C.

davidemyers · 2019-08-18T13:26:15Z

A cloud deployment from macOS 10.14.6 to DigitalOcean using python3 and LANG=C.UTF-8 also works.

Since I already had Homebrew installed I set up the dependencies like this:

brew install python
python3 -m pip install --upgrade virtualenv # Don't need --user

I think when the time comes to move to python3 the instructions for macOS users might want to include both using Homebrew and installing the official Python 3 package as options. Homebrew is convenient but if you're not already using it it's kind of a big dependency. Though I wonder if the official Python 3 package will require updating the certificate store like the Python 2 package does, as discussed in #1545.

Edited to add one more note about Python 3: When running a local or DO deploy on Ubuntu the only prerequisite package I needed to install was python3-virtualenv. That's a considerable reduction in required packages for those doing cloud deploys.

jackivanov · 2019-08-19T09:37:45Z

@davidemyers I think we need to address python3 in a separate issue

davidemyers · 2019-08-19T11:28:01Z

@jackivanov I agree, I was just curious to know what already worked and thought I'd record what I found in case it's useful later.

jackivanov · 2019-08-22T10:16:53Z

Actually, I'd like to switch to python3 in this pr, so still work in progress

davidemyers · 2019-08-22T13:05:58Z

I'm not a Python expert, but I think PEP 538 explains the issue with generating the p12 password in Python 3. This works at the command line on 18.04, 19.04, and macOS:

openssl rand 8 | LANG=C python3 -c 'import sys,string; chars=string.ascii_letters + string.digits + "_@"; print("".join([chars[ord(c) % 64] for c in list(sys.stdin.read())]))'

I haven't tried it this way yet when invoked from Ansible.

davidemyers · 2019-08-23T13:14:16Z

algo needs env -> .env. After being reverted, I guess.

bhawkins · 2019-08-31T22:05:35Z

I just installed from this branch on WSL, though I had to patch an Ansible script:

$ diff -Naur .env/lib/python3.5/site-packages/ansible/modules/cloud/digital_ocean/di
gital_ocean_sshkey.py{.orig,}
--- .env/lib/python3.5/site-packages/ansible/modules/cloud/digital_ocean/digital_ocean_sshkey.py.orig   2019-08-31 15:01:22.606761500 -0700
+++ .env/lib/python3.5/site-packages/ansible/modules/cloud/digital_ocean/digital_ocean_sshkey.py        2019-08-31 15:02:23.758253300 -0700
@@ -107,7 +107,7 @@
                 return json.loads(self.info["body"])
             return None
         try:
-            return json.loads(self.body)
+            return json.loads(self.body.decode())
         except ValueError:
             return None

jackivanov · 2019-09-02T09:59:18Z

It appears that the minimum required version of python should be 3.6, and it seems like ubuntu 16.04 is incompatible now

davidemyers · 2019-09-03T12:27:58Z

You should also remove the sentence that starts with "On Ubuntu 16.04 LTS ..." that I recently added to the README.

davidemyers · 2019-09-06T12:33:10Z

This branch works again without library/digital_ocean_droplet.py.

docs/deploy-from-script-or-cloud-init-to-localhost.md

btonic · 2019-09-20T23:59:29Z

library/gcp_compute_location_info.py

+    else:
+        items = []
+    return_value = {'resources': items}
+    module.exit_json(**return_value)


Could this be simplified to the following?:

module.exit_json(**dict(resources=items.get('items', [])))

btonic · 2019-09-21T00:17:02Z

library/gcp_compute_location_info.py

+    return return_if_object(module, response)
+
+
+def query_options(filters):


I think we could make this a little more concise with list comprehensions:

def query_options(filters): if not filters: return '' elif len(filters) == 1: return filters[0] else: return ' '.join( filter if (filter.startswith('(') and filter.endswith(')')) else f"({filter})" for filter in filters )

Actually, the first two cases are implicitly handled if we turn it into a generator:

def query_options(filters): return ' '.join( filter if (filter.startswith('(') and filter.endswith(')')) else f"({filter})" for filter in filters )

btonic · 2019-09-21T00:34:36Z

library/gcp_compute_location_info.py

+
+def return_if_object(module, response):
+    # If not found, return nothing.
+    if response.status_code == 404:


we can merge these together:

if response.status_code in (404, 204,): return None

reaperhulk

Thanks for your contribution! I have sanity checked this patch and tried to look through it. To the extent that it's possible to review this looks okay, but in the future it would be much easier to do a thorough review job without putting so many distinct changes in a PR.

Ideally each of the changes listed in this PR would have been a separate PR, so instead of one 2000 line diff we'd be looking at ~8 separate PRs, each with manageable diff sizes.

@btonic provided some review feedback that it would be nice to see in a followup PR as well, but for now we'll merge the changes. Thanks again for your work!

* bump ansible to 2.8.3 * DigitalOcean: move to the latest modules * Add Hetzner Cloud * Scaleway and Lightsail fixes * lint missing roles * Update roles/cloud-hetzner/tasks/main.yml Add api_token Co-Authored-By: phaer <[email protected]> * Update roles/cloud-hetzner/tasks/main.yml Add api_token Co-Authored-By: phaer <[email protected]> * Try to run apt until succeeded * Scaleway modules upgrade * GCP: Refactoring, remove deprecated modules * Doc updates (#1552) * Update README.md Adding links and mentions of Exoscale aka CloudStack and Hetzner Cloud. * Update index.md Add the Hetzner Cloud to the docs index * Remove link to Win 10 IPsec instructions * Delete client-windows.md Unnecessary since the deprecation of IPsec for Win10. * Update deploy-from-ansible.md Added sections and required variables for CloudStack and Hetzner Cloud. * Update deploy-from-ansible.md Added sections for CloudStack and Hetzner, added req variables and examples, mentioned environment variables, and added links to the provider role section. * Update deploy-from-ansible.md Cosmetic changes to links, fix typo. * Update GCE variables * Update deploy-from-script-or-cloud-init-to-localhost.md Fix a finer point, and make variables list more readable. * update azure requirements * Python3 draft * set LANG=c to the p12 password generation task * Update README * Install cloud requirements to the existing venv * FreeBSD fix * env->.env fixes * lightsail_region_facts fix * yaml syntax fix * Update README for Python 3 (#1564) * Update README for Python 3 * Remove tabs and tweak instructions * Remove cosmetic command indentation * Update README.md * Update README for Python 3 (#1565) * DO fix for "found unpermitted parameters: id" * Verify Python version * Remove ubuntu 16.04 from readme * Revert back DigitalOcean module * Update deploy-from-script-or-cloud-init-to-localhost.md * env to .env

jackivanov added the needs_tests label Aug 15, 2019

phaer reviewed Aug 16, 2019

View reviewed changes

roles/cloud-hetzner/tasks/main.yml Outdated Show resolved Hide resolved

roles/cloud-hetzner/tasks/main.yml Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

jackivanov changed the title ~~Bump Ansible to 2.8 and new provider Hetzner Cloud~~ [WIP] Bump Ansible to 2.8 and new provider Hetzner Cloud Aug 16, 2019

jackivanov changed the title ~~[WIP] Bump Ansible to 2.8 and new provider Hetzner Cloud~~ Large refactor to support Ansible 2.8 Aug 21, 2019

jackivanov changed the title ~~Large refactor to support Ansible 2.8~~ Refactor to support Ansible 2.8 Aug 21, 2019

jackivanov added ready_to_merge and removed needs_tests ready_to_merge labels Aug 21, 2019

jackivanov changed the title ~~Refactor to support Ansible 2.8~~ [WIP] Refactor to support Ansible 2.8 Aug 22, 2019

jackivanov force-pushed the ansible-2.8 branch 2 times, most recently from 476420e to ec1c319 Compare August 22, 2019 15:33

bump ansible to 2.8.3

866149a

squirrel532 mentioned this pull request Sep 1, 2019

use openssl rand -base64 to generate password #1574

Closed

3 tasks

Verify Python version

ba2a4a0

Remove ubuntu 16.04 from readme

c5a576c

jackivanov mentioned this pull request Sep 6, 2019

Fix macOS installation guide #1542

Closed

jackivanov added this to the 1.2 milestone Sep 6, 2019

jackivanov added the 1.2 label Sep 6, 2019

Merge branch 'master' into ansible-2.8

a2da4b6

TC1977 reviewed Sep 6, 2019

View reviewed changes

docs/deploy-from-script-or-cloud-init-to-localhost.md Outdated Show resolved Hide resolved

jackivanov and others added 2 commits September 8, 2019 10:56

Revert back DigitalOcean module

b39a2e6

Update deploy-from-script-or-cloud-init-to-localhost.md

c9aa372

jackivanov changed the title ~~[WIP] Refactor to support Ansible 2.8~~ Refactor to support Ansible 2.8 Sep 10, 2019

jackivanov requested review from dguido and removed request for dguido September 10, 2019 08:30

jackivanov added 2 commits September 12, 2019 12:15

Merge branch 'master' into ansible-2.8

b8f0cd5

env to .env

56e6099

jackivanov mentioned this pull request Sep 18, 2019

Can't run ./algo, Abort trap: 6 #1539

Closed

btonic reviewed Sep 20, 2019

View reviewed changes

btonic reviewed Sep 21, 2019

View reviewed changes

squirrel532 mentioned this pull request Sep 21, 2019

Linode support #1497

Closed

reaperhulk approved these changes Sep 28, 2019

View reviewed changes

reaperhulk merged commit 8bdd99c into master Sep 28, 2019

dguido deleted the ansible-2.8 branch September 28, 2019 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor to support Ansible 2.8 #1549

Refactor to support Ansible 2.8 #1549

jackivanov commented Aug 15, 2019 •

edited

Loading

davidemyers commented Aug 15, 2019

phaer left a comment •

edited

Loading

jackivanov commented Aug 16, 2019

phaer commented Aug 16, 2019

This comment has been minimized.

davidemyers commented Aug 16, 2019

davidemyers commented Aug 16, 2019 •

edited

Loading

TC1977 commented Aug 16, 2019

jackivanov commented Aug 16, 2019

TC1977 commented Aug 16, 2019 •

edited

Loading

jackivanov commented Aug 16, 2019

TC1977 commented Aug 16, 2019

davidemyers commented Aug 17, 2019 •

edited

Loading

davidemyers commented Aug 18, 2019 •

edited

Loading

jackivanov commented Aug 19, 2019

davidemyers commented Aug 19, 2019

jackivanov commented Aug 22, 2019

davidemyers commented Aug 22, 2019

davidemyers commented Aug 23, 2019

bhawkins commented Aug 31, 2019

jackivanov commented Sep 2, 2019 •

edited

Loading

davidemyers commented Sep 3, 2019

davidemyers commented Sep 6, 2019

btonic Sep 20, 2019

btonic Sep 21, 2019

btonic Sep 21, 2019

btonic Sep 21, 2019

reaperhulk left a comment

		return return_if_object(module, response)


		def query_options(filters):

Refactor to support Ansible 2.8 #1549

Refactor to support Ansible 2.8 #1549

Conversation

jackivanov commented Aug 15, 2019 • edited Loading

Upgrades and Fixes

Cloud provider changes

Breaking changes

Types of changes

Checklist:

davidemyers commented Aug 15, 2019

phaer left a comment • edited Loading

Choose a reason for hiding this comment

jackivanov commented Aug 16, 2019

phaer commented Aug 16, 2019

This comment has been minimized.

davidemyers commented Aug 16, 2019

davidemyers commented Aug 16, 2019 • edited Loading

TC1977 commented Aug 16, 2019

jackivanov commented Aug 16, 2019

TC1977 commented Aug 16, 2019 • edited Loading

jackivanov commented Aug 16, 2019

TC1977 commented Aug 16, 2019

davidemyers commented Aug 17, 2019 • edited Loading

davidemyers commented Aug 18, 2019 • edited Loading

jackivanov commented Aug 19, 2019

davidemyers commented Aug 19, 2019

jackivanov commented Aug 22, 2019

davidemyers commented Aug 22, 2019

davidemyers commented Aug 23, 2019

bhawkins commented Aug 31, 2019

jackivanov commented Sep 2, 2019 • edited Loading

davidemyers commented Sep 3, 2019

davidemyers commented Sep 6, 2019

btonic Sep 20, 2019

Choose a reason for hiding this comment

btonic Sep 21, 2019

Choose a reason for hiding this comment

btonic Sep 21, 2019

Choose a reason for hiding this comment

btonic Sep 21, 2019

Choose a reason for hiding this comment

reaperhulk left a comment

Choose a reason for hiding this comment

jackivanov commented Aug 15, 2019 •

edited

Loading

phaer left a comment •

edited

Loading

davidemyers commented Aug 16, 2019 •

edited

Loading

TC1977 commented Aug 16, 2019 •

edited

Loading

davidemyers commented Aug 17, 2019 •

edited

Loading

davidemyers commented Aug 18, 2019 •

edited

Loading

jackivanov commented Sep 2, 2019 •

edited

Loading