Ns web vm deployment #255

kennethgillen · 2017-03-24T16:38:22Z

Deployment of https://trello.com/c/vxPG2L3a/714-new-nightshade-web-vm

Configuration in management tools at https://github.com/openmicroscopy/management_tools/pull/352

Still in development

kennethgillen · 2017-03-30T13:12:16Z

This version run yesterday to provision ns-web-vm.

LS27172:nightshade-web kenny$ echorun ansible-playbook -l ns-web.openmicroscopy.org -i ~/Forks/management_tools/ansible/inventory/prod-hosts --ask-become-pass playbook.yml | tee ns-web-provision_$(date -u +"%Y%m%d-%H%M%SZ").log

Run:
ns-web-provision_20170329-150136Z.txt

Run (after removing yum exclusions for check-mk):
ns-web-provision_20170329-152200Z.txt

kennethgillen · 2017-04-07T15:46:27Z

This version run a few minutes ago, installing SSL config and certificate.

LS27172:nightshade-web kenny$ echorun ansible-playbook --diff -l ns-web.openmicroscopy.org -i ~/Forks/management_tools/ansible/inventory/prod-hosts --ask-become-pass --ask-vault-pass playbook.yml | tee ns-web-p
rovision_$(date -u +"%Y%m%d-%H%M%SZ").txt
# Fri  7 Apr 2017 16:41:23 BST:  ansible-playbook --diff -l ns-web.openmicroscopy.org -i /Users/kenny/Forks/management_tools/ansible/inventory/prod-hosts --ask-become-pass --ask-vault-pass playbook.yml

Dependent config in private repo https://github.com/openmicroscopy/management_tools/pull/352

ns-web-provision_20170407-154123Z.txt

https://ns-web.openmicroscopy.org/webgateway/img_detail/3933597/

kennethgillen · 2017-04-10T14:03:36Z

(temporary) manual configuration of ns-web performed installs the web-extensions to bring ns-web to feature parity with Nightshade. Server ready for testing. Not added to the playbook, as it's changes that won't be necessary with 5.3.0. Can be added later if required.

…agging

kennethgillen · 2017-04-12T13:31:02Z

Now added the 'manual' install steps for figure and webtagging, as mentioned in #255 (comment), as I want to use the Ansible to push more config settings, so require it to continue to work.

manics · 2017-04-12T13:53:08Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+
+  handlers:
+
+  # Handler for nginx


The omero-common role already includes a set of useful handlers, you should be able to use that instead of redefining one here.

👍 - I didn't realise I could use things defined 'underneath' the playbook when I wrote that. From programming, I wouldn't have expected global scope for defined functions like that. I subsequently used e.g. the 'restart systemd' handler too, so you can see I learned, but didn't strip the original nginx one from the playbook.

manics · 2017-04-12T13:54:00Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+        destfile: /etc/systemd/system/omero-web.service.d/pythonpath.conf
+        owner: "root" 
+        group: "root" 
+        block: |2+


Do you need the fancy formatting provided by |2+? Will just plain | do?

I couldn't make it work any other way, so leaving it as 'this works'. Other options failed completely with errors, or didn't give the indenting in the destination. I am pretty new to YAML and the whitespace significance, though.

manics · 2017-04-12T13:54:54Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+      file:
+        path: "{{ omero_web_extensionsdir }}"
+        state: directory
+        owner: "{{ omero_web_system_user }}" 


Does this directory need to be writeable by the web user? And similarly for the tasks below.

Not sure, but this works, and it's how it's set up elsewhere. If you've another suggestion, happy to hear it.

Make it owned by root unless you have reason to believe OMERO.web needs write access to its code.

Let's try it.

manics · 2017-04-12T13:56:20Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+      template:
+        src: templates/omero-web-config-for-webapps.j2
+        dest: "{{ omero_web_basedir }}/config/omero-web-config-for-webapps.omero"
+        owner: "{{ omero_web_system_user }}" 


Does this need to be writeable by web?

Set now to root:root.

manics · 2017-04-12T13:57:02Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+        group: "{{ omero_web_system_user }}" 
+        mode: "u=rw,go=r"
+      notify:
+        - omero-web restart omero-web 


Can you use the handler from omero-common restart omero-web instead of the one embedded in the omero-web role which was intended for internal use by the role?

¯_(ツ)_/¯ I just grep -R restart * and found this in use elsewhere and having learned it's possible to re-use them, I used it. I can try.

Yep, seems to work

changed: [infra-testpr.openmicroscopy.org] RUNNING HANDLER [openmicroscopy.omero-common : restart omero-web] ************** changed: [infra-testpr.openmicroscopy.org] PLAY RECAP ********************************************************************* infra-testpr.openmicroscopy.org : ok=3 changed=2 unreachable=0 failed=0

manics · 2017-04-12T13:58:29Z

ansible/server-state-playbooks/nightshade-web/templates/omero-web-config-for-webapps.j2

+
+config set -- omero.web.apps '["autotag", "tagsearch", "figure"]'
+config set -- omero.web.ui.center_plugins '[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]'
+config set -- omero.web.ui.top_links '[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Figure", "figure_index", {"target": "new", "title": "Open OMERO.Figure in a new tab"}], ["Tag Search", "tagsearch"]]'


Minor point: there's no significant templated variables in here, so it could be a plain file (task copy instead of a template)

👍 will consider it for next time.

kennethgillen · 2017-04-12T15:50:39Z

Current 7003b1e successfully working on infra-testpr, figure generation and tagsearch both AOK.

PLAY RECAP *********************************************************************
infra-testpr.openmicroscopy.org : ok=65   changed=37   unreachable=0    failed=0

manics

A couple of non-blocker comments. Looks good to me.

manics · 2017-04-12T15:55:36Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+        owner="{{ item.value.owner }}"
+        group="{{ item.value.group }}"
+        mode="{{ item.value.mode }}"
+      with_dict: "{{ nginx_ssl_cert_files }}"


with_dict looks a bit weird here, though I haven't seen the actual variable.

@manics - see https://dantehranian.wordpress.com/2015/07/24/managing-secrets-with-ansible-vault-the-missing-guide-part-1-of-2/ - it's exactly like this example.

OK, looks overly complicated since the owner/group/mode don't vary

manics · 2017-04-12T15:57:47Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

+    - name: OMERO.web apps | download latest figure
+      become: yes
+      unarchive:
+        src: https://downloads.openmicroscopy.org/latest/figure.zip


For full reproducibility you should use the versioned zip https://downloads.openmicroscopy.org/figure/1.2.1/figure-1.2.1.zip since you've got creates: "{{ omero_web_extensionsdir }}/figure-{{omero_web_extensions_figure_ver}}" below

manics · 2017-04-12T15:59:29Z

ansible/server-state-playbooks/nightshade-web/requirements.yml

+---
+
+- name: openmicroscopy.omero-common
+  src:  https://github.com/openmicroscopy/ansible-role-omero-common.git


Remember to add versions once these have been tagged.

👍 see workaround in ome/pydoop-features@fec18e1 until we have tags being properly pushed to galaxy again (ansible/galaxy-issues#252)

Alternatively since you've got git installed on the machine that's running Ansible you can just add version: X.X.X, e.g. IDR/deployment@d74d7c3
The main advantage of ome/pydoop-features@fec18e1 is you don't need to install git.

…want to leak config to ansible logfiles

…mmunity/tutorials/how-to-optimize-nginx-configuration

kennethgillen · 2017-04-12T17:43:54Z

ns-web reset to 'post provision' snapshot, and redeployed this playbook at current version, combined with private config at 11901be

ns-web-provision_20170412-173859Z.txt
ns-web-provision_20170412-171603Z.txt

…iply with #processors to get total cores

kennethgillen · 2017-04-13T10:19:27Z

re-ran with updated 'total cores' math (matching private config https://github.com/openmicroscopy/management_tools/pull/352/commits/9292f47b294acb09d6e043fe626ae45237357113)

ns-web-provision_20170413-101658Z.txt

PLAY RECAP *********************************************************************
ns-web.openmicroscopy.org  : ok=59   changed=4    unreachable=0    failed=0

kennethgillen added 3 commits March 23, 2017 20:45

Initial OMERO web-only

d6a1d6f

LVM commented out for testing, but progressing well.

b45aea4

Additional check-mk

0a4d389

joshmoore added the ON_HOLD label Mar 29, 2017

Run configured for SLS VM versus local vagrant

0028765

kennethgillen added 2 commits April 6, 2017 19:40

nginx SSL transform added from OMERO.web generated file

44b6232

nginx restart handler, plus SSL configuration

bc8330f

kennethgillen removed the ON_HOLD label Apr 10, 2017

kennethgillen added 2 commits April 12, 2017 14:22

Manual, i.e. non-pip install of omero web extensions/apps figure an t…

3a23209

…agging

ignore the roles folder which are pulled down from galaxy

75395a9

manics suggested changes Apr 12, 2017

View reviewed changes

kennethgillen added 3 commits April 12, 2017 16:10

Addressing @manics concerns

71e4348

Addressing @manics concerns

0eb8e96

root:root as advised by @manics via GH code review

7003b1e

manics approved these changes Apr 12, 2017

View reviewed changes

kennethgillen added 2 commits April 12, 2017 17:05

adding no_log because it's not set in the omero-web role and I don't …

f8be358

…want to leak config to ansible logfiles

Tuning nginx workers/processes as per https://www.digitalocean.com/co…

b0dabca

…mmunity/tutorials/how-to-optimize-nginx-configuration

ansible_processor_cores was not total, but per-processor. Had to mult…

bef823f

…iply with #processors to get total cores

sbesson merged commit e75bdc7 into ome:master Apr 13, 2017

kennethgillen deleted the ns-web-vm branch April 13, 2017 10:50

joshmoore mentioned this pull request Dec 1, 2017

Review infra PRs for modifications ome/prod-playbooks#16

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ns web vm deployment #255

Ns web vm deployment #255

kennethgillen commented Mar 24, 2017

kennethgillen commented Mar 30, 2017

kennethgillen commented Apr 7, 2017 •

edited

Loading

kennethgillen commented Apr 10, 2017

kennethgillen commented Apr 12, 2017

manics Apr 12, 2017

kennethgillen Apr 12, 2017 •

edited

Loading

manics Apr 12, 2017

kennethgillen Apr 12, 2017 •

edited

Loading

manics Apr 12, 2017

kennethgillen Apr 12, 2017

manics Apr 12, 2017 •

edited

Loading

kennethgillen Apr 12, 2017

manics Apr 12, 2017

kennethgillen Apr 12, 2017

manics Apr 12, 2017

kennethgillen Apr 12, 2017 •

edited

Loading

kennethgillen Apr 12, 2017

manics Apr 12, 2017

kennethgillen Apr 12, 2017

kennethgillen commented Apr 12, 2017

manics left a comment

manics Apr 12, 2017

kennethgillen Apr 19, 2017

manics Apr 19, 2017

manics Apr 12, 2017

manics Apr 12, 2017

joshmoore Apr 19, 2017

manics Apr 19, 2017 •

edited

Loading

kennethgillen commented Apr 12, 2017

kennethgillen commented Apr 13, 2017 •

edited

Loading

Ns web vm deployment #255

Ns web vm deployment #255

Conversation

kennethgillen commented Mar 24, 2017

kennethgillen commented Mar 30, 2017

kennethgillen commented Apr 7, 2017 • edited Loading

kennethgillen commented Apr 10, 2017

kennethgillen commented Apr 12, 2017

Choose a reason for hiding this comment

kennethgillen Apr 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennethgillen Apr 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manics Apr 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennethgillen Apr 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennethgillen commented Apr 12, 2017

manics left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manics Apr 19, 2017 • edited Loading

Choose a reason for hiding this comment

kennethgillen commented Apr 12, 2017

kennethgillen commented Apr 13, 2017 • edited Loading

kennethgillen commented Apr 7, 2017 •

edited

Loading

kennethgillen Apr 12, 2017 •

edited

Loading

kennethgillen Apr 12, 2017 •

edited

Loading

manics Apr 12, 2017 •

edited

Loading

kennethgillen Apr 12, 2017 •

edited

Loading

manics Apr 19, 2017 •

edited

Loading

kennethgillen commented Apr 13, 2017 •

edited

Loading