Rolling issue: update central installations on CEDA/Jasmin and DKRZ

[valeriupredoi]

List of installed versions

• 28/20/2020 CEDA/Jasmin ESMValCore/ESMValTool=2.1.0
• 03/12/2020 CEDA/Jasmin ESMValCore=2.1.0/ESMValTool=2.1.1

Guff

• we should update the central installs on Jasmin and DRKZ with the latest version 👍
• EDIT 28/10/2020 - this is done on Jasmin; log in /apps/jasmin/community/esmvaltool/install_logs
  Summary: example recipes (python, ncl, julia and r) run fine, tests:

pytest
...
1 failed, 1133 passed, 1 skipped, 4 warnings in 411.67s (0:06:51)

test failed documented in #1890

• remember the Python3.9 transition!

[valeriupredoi]

Here is an example of a workflow to install eg 2.1.1:

Steps to reproduce the installation of ESMValTool v2.1.1:
(NOTE: this assumes a miniconda3 setup already being done
for a previous ESMValTool version)

source conda_base.sh
git clone https://github.com/ESMValGroup/ESMValTool.git ESMValTool_2.1.1
conda -V (4.9.1)
conda update conda (to update to 4.9.2)
conda -V (4.9.2)
mv miniconda3/envs/esmvaltool/ miniconda3/envs/esmvaltool-2.1.0
cd ESMValTool_2.1.1
conda env create -n esmvaltool -f environment.yml
conda activate esmvaltool
esmvaltool version
ln -s /apps/jasmin/community/esmvaltool/julia-1.0.3/bin/julia /apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin/julia
which julia
julia --version (1.0.3)
pip install -e .[develop]
esmvaltool version
esmvaltool --help
esmvaltool install Julia
esmvaltool install R
pytest -m "not installation"
pytest

and the conda_base.sh is

eval "$(/apps/jasmin/community/esmvaltool/miniconda3/bin/conda shell.bash hook)"

then the module is built with a shell file:

export PATH=/apps/jasmin/community/esmvaltool/miniconda3/bin:$PATH
export PATH=/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages:$PATH
export PATH=/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/bin:$PATH

[bouweandela]

Just saw this issue. Central installations on both systems appear to be outdated/not available

• Mistral is about to be decommissioned (end of the month) so probably doesn't matter, but was never updated beyond v2.2
• Levante does not have a central installation yet
• Jasmin is at v2.4, while the latest release is v2.5

[valeriupredoi]

which JASMIN you on, Bouwe?

[valeriu@sci2 ~]$ module load esmvaltool
[valeriu@sci2 ~]$ esmvaltool version
ESMValCore: 2.5.0
ESMValTool: 2.5.0

(please don't tell me you travelled in time to pre-pandemic/pre-war in Ukr times 😁 )

[remi-kazeroni]

Just saw this issue. Central installations on both systems appear to be outdated/not available

* Mistral is about to be decommissioned (end of the month) so probably doesn't matter, but was never updated beyond v2.2

* Levante does not have a central installation yet

* Jasmin is at v2.4, while the latest release is v2.5

You're right about the situation at DKRZ. On Mistral, it has not been possible to obtain regular updates of the module as we wished. Since Mistral will be soon offline (end of the month), I agree it doesn't make sense to try again.

I haven't tried to contact DKRZ about getting a module on Levante because I heard that DKRZ staff is very busy getting the machine up to full speed. Many basic modules are still missing, I guess esmvaltool won't be a priority now. My idea was to wait until the next release (v2.6) to contact DKRZ and find a way to obtain a regularly updated module. If that is still not possible on Levante, we could try to provide our own shared installation and document it in our docs. Not as nice as a module but at still we would have control on the installation. Would that sound ok?

[bouweandela]

@valeriupredoi You're right, but the name of the module still contains 2.4?

bandela@sci2:~$ module avail esmvaltool

-------------------------------------------------------------------------------------------------------------------------------- /apps/jasmin/modulefiles --------------------------------------------------------------------------------------------------------------------------------
esmvaltool/2.4(default)
bandela@sci2:~$ module load esmvaltool
bandela@sci2:~$ esmvaltool version
ESMValCore: 2.5.0
ESMValTool: 2.5.0

[valeriupredoi]

alas that is true! @agstephens please help and rename the module from /apps/jasmin/modulefiles/esmvaltool/2.4 to /apps/jasmin/modulefiles/esmvaltool/2.5 🙏 (will mssg on Slack too)

[zklaus]

Perhaps it would be worthwhile keeping a table or list of these installations on the new webpage?

[remi-kazeroni]

Perhaps it would be worthwhile keeping a table or list of these installations on the new webpage?

We have it there. Did you have another webpage in mind?

[bouweandela]

@valeriupredoi While you're at it, the (link to the) logo on this help page is also broken: https://help.jasmin.ac.uk/article/4955-community-software-esmvaltool

[bouweandela]

If that is still not possible on Levante, we could try to provide our own shared installation and document it in our docs. Not as nice as a module but at still we would have control on the installation. Would that sound ok?

I think it would be better if the system admins on Levante made the module available. We can offer to help them with that if it is complicated. If they need another type of packaging or other improvements to make it easier for them to make a module, we could work on those.

[zklaus]

Perhaps it would be worthwhile keeping a table or list of these installations on the new webpage?

We have it there. Did you have another webpage in mind?

I was thinking about the main web page currently being built with Jekyll if I am not mistaken. The reason to keep it outside of the documentation is that it should name specifically the versions available, whereas the documentation always pertains to one specific version. I think it would be nice to have a short table that says which versions are currently supported on which HPC (with the expectation that at any given time more than one version is available).

[remi-kazeroni]

If that is still not possible on Levante, we could try to provide our own shared installation and document it in our docs. Not as nice as a module but at still we would have control on the installation. Would that sound ok?

I think it would be better if the system admins on Levante made the module available. We can offer to help them with that if it is complicated. If they need another type of packaging or other improvements to make it easier for them to make a module, we could work on those.

I've had an email discussion with some people at DKRZ and I can report there is willingness to have a module and a Jupyter kernel for ESMValTool for Levante. Not sure about the timeline for implementation but I'll let you know as soon as it is done.

[remi-kazeroni]

Since today we have a module on Levante at DKRZ, see:

(base) [Tue Jun 28 16:24:13] b309192@levante2:~
 $ module load esmvaltool

(base) [Tue Jun 28 16:24:18] b309192@levante2:~
 $ esmvaltool version
ESMValCore: 2.5.0
ESMValTool: 2.5.0

I will continue my discussions with DKRZ to get a Jupyter kernel associated to this module. Also, I'll see with them how to update this module after our releases in a way that wouldn't require too much work for DKRZ.

[valeriupredoi]

wohoo! Great work @remi-kazeroni 🍺

[valeriupredoi]

now if you write a bit docs we can close this forevah 😁

[remi-kazeroni]

now if you write a bit docs we can close this forevah 😁

Done in #2696 😄 But in fact, we should also update the documentation for the Core once #2696 is merged.

[valeriupredoi]

cheers Remi! Indeed we should 👍