WIP: Update sn07 playbook and the requirements.yml file

[sanjaysrikakulam]

Final changes made to sn07.yml after testing the playbook manually against the sn07 server.

File sn07.yml:

1. galaxyproject.gxadmin: Missing make dependency for galaxyproject.gxadmin role. So added it through a PR to the usegalaxy_eu.handy role.
2. galaxyproject.tiaas2:
   1. This role should be moved after the second invocation of galaxyproject.galaxy role; only during the second invocation the variable galaxy_manage_clone is set to true and the tiaas2 role requires this cloned repo to accomplish the task Copy Galaxy's stylesheet otherwise ansible will fail, so the role was moved after the second invocation of the galaxyproject.galaxy role.
   2. Since we use the virtualenv from a conda environment we should add the variable tiaas_virtualenv_command directly under this role. Adding the variable to the group_vars/sn07.yml would not work due to variable precedence. Also, this variable should point to the virtualenv present in the conda _galaxy_ environment. This environment is created by the galaxyproject.miniconda role. All virtual environments are created using the virtualenv command from this _galaxy_ environment
3. dj-wasabi.telegraf: This role should run before the usegalaxy-eu.galaxy-procstat role else the role will fail because it cannot find the /etc/telegraf/telegraf.d directory (This directory will get created only after the installation of telegraf package).
4. dev-sec.ssh-hardening:
   1. Is not maintained anymore. Instead, we should use their ansible-collection-hardening collection.
   2. Rocky 9 uses OpenSSH version 8.7 and does not support [email protected] KEX algorithm which has been fixed in the above-mentioned collection.

File requirements.yml:

1. Updated the version of the collection devsec.hardening to 8.3.0 because an SELinux task in the ssh_hardening role fails. The updated collection fixes that issue.

Manual changes made on the server to make galaxy run:

1. Change ownership of the file /opt/galaxy/server/compliance.log to the user galaxy
2. SELinux set to permissive mode and/or disable (just like in sn06)
3. Stop firewalld service and disable it (just like in sn06)
4. Setting the systemd variable galaxy_systemd_handlers to 0 in group_vars/sn07.yml results in a broken Job_conf.yml because the processes section in the handling block is empty and this leads to a NoneType error which stops galaxy from starting. So manually adding a dummy handler process name would allow the galaxy processes to start. This is a non-persistent change and this file will be overwritten during the next Ansible run via Jenkins.

Attach this PR to: https://github.com/usegalaxy-eu/issues/issues/352

[bgruening]

Itsn't this installing TIaaS into the virtual env of Galaxy?

[sanjaysrikakulam]

No, galaxy_virtualenv_command contains only the path to the virtualenv command present in a conda environment. TIaaS env is created as usual inside /opt/tiaas2.

[bgruening]

Setting the systemd variable galaxy_systemd_handlers to 0

This one looks like we could be smarter in the template to allow this use-case.

/opt/galaxy/server/compliance.log

Do you know who created this file?

[mira-miracoli]

This one looks like we could be smarter in the template to allow this use-case.

I think its a galaxy problem, it refuses to start when there is no handler entry in the yaml.

[mira-miracoli]

Looks good to me

[sanjaysrikakulam]

Setting the systemd variable galaxy_systemd_handlers to 0

This one looks like we could be smarter in the template to allow this use-case.

This is the job_handlers block in the job_conf.yml

#
# Job handler configuration
#
handling:
  assign_with:
    - db-skip-locked
  max_grab: 16
  ready_window_size: 32
  processes:
    dummy_test_sn07_h0:

As part of my testing, I tried to comment out the processes section and restarted the web handlers and found various errors in the logs, then attempted to comment out the entire handling block then restarted the web handlers and I still found different errors. I was able to get this to work only after adding a dummy process name.

I was testing this to find out how to modify the templating to handle 0 handlers but could not figure out how the galaxy works things out.

/opt/galaxy/server/compliance.log

Do you know who created this file?

This file is probably created by a logging module and I was able to find that this is present in this python file /opt/galaxy/server/lib/galaxy/config/__init__.py. Though this compliance.log is empty galaxy still needs the write access on the file to start the galaxy server processes other it fails.

[sanjaysrikakulam]

WIP please do not merge.

[sanjaysrikakulam]

• SELinux set to permissive mode and/or disable (just like in sn06)
• Stop firewalld service and disable it (just like in sn06)

@bgruening I have added the relevant tasks for the above mentioned manual changes. With these changes, we do not have to disable SELinux and firewalld.

[sanjaysrikakulam]

• Change ownership of the file /opt/galaxy/server/compliance.log to the user galaxy
• Setting the systemd variable galaxy_systemd_handlers to 0 in group_vars/sn07.yml results in a broken Job_conf.yml because the processes section in the handling block is empty and this leads to a NoneType error which stops galaxy from starting. So manually adding a dummy handler process name would allow the galaxy processes to start. This is a non-persistent change and this file will be overwritten during the next Ansible run via Jenkins.

With the latest commit the above two manual changes are also now automated. Along with that the corn job Gxadmin Galaxy clean up is also disabled on sn07. The reason for disabling is that we do not want to run this cron job from both the head nodes.

[mira-miracoli]

looks good to me