Normalize input units prior to NCL unit conversion

[schlunma]

Description

Normalize units (i.e., replace "." with " " and sort them) prior to unit conversion. This allows NCL to parse units like m-2.kg.s-1 (which are normalized to kg m-2 s-1).

• Closes Conversion unit issues in diagnostic used in recipe_wenzel14jgr.yml #3224

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

• 🧪 Recipe runs successfully
• 🧪 Recipe is well documented
• 🧪 Figure(s) and data look as expected from literature
• 🛠 Provenance information has been added

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[remi-kazeroni]

Thanks @schlunma! After some struggle with data not findable on ESGF (had to change some settings in my esgf-pyclient.yml), I was able to run the recipe successfully. Here is the run directory: /scratch/b/b309192/runs/tests/esmvaltool_output/recipe_wenzel14jgr_20230619_090924.

@axel-lauer - could you please have a look at the NCL code and merge this PR if all all looks good to you?

[axel-lauer]

The code looks good, but I am not sure I like how the problem of the units string is fixed. It is fine to replace the periods "." with spaces in the units string but I find it strange to sort of components of the units string alphabetically. While this is fine in case of "kg m-2 s-1", e.g. "W.m-2" would result in "m-2 W", which I find strange.

The name of the function "normalize_units" sounds very general, suggesting that this fix should work for all cases. This would make it very hard to trace future problems with this fix back to this function. If really needed, I would therefore prefer to hard code this fix specifically for "m-2.kg.s-1" instead. The best option, however, would be to fix the actual problem, which I think is coming from ESMValCore.

I experimented a bit, this is what I found out:

• ESMValCore 2.8 does not change the units attribute (OK).
• ESMValCore 2.9 with a v2.8 conda environment does not change the units attribute (OK).
• ESMValCore 2.9 with a v2.9 conda environment changes the units attribute (ERROR)

I could trace back the problem of the broken units attribute in the preprocessed files to the preprocessor "area_statistics " (operator sum). I would therefore expect, all recipes using this preprocessor might also be affected, even though this probably shows in weird labels in the plot only.

Given the tests I did, I would think this is a problem of a changed dependency of the core and should probably be best addressed in the core directly.

[valeriupredoi]

OK boomer (reference to NCL, not Manu who is too young to be called "boomer") 🍺

[valeriupredoi]

The code looks good, but I am not sure I like how the problem of the units string is fixed. It is fine to replace the periods "." with spaces in the units string but I find it strange to sort of components of the units string alphabetically. While this is fine in case of "kg m-2 s-1", e.g. "W.m-2" would result in "m-2 W", which I find strange.

The name of the function "normalize_units" sounds very general, suggesting that this fix should work for all cases. This would make it very hard to trace future problems with this fix back to this function. If really needed, I would therefore prefer to hard code this fix specifically for "m-2.kg.s-1" instead. The best option, however, would be to fix the actual problem, which I think is coming from ESMValCore.

I experimented a bit, this is what I found out:

* ESMValCore 2.8 does not change the units attribute (OK).

* ESMValCore 2.9 with a v2.8 conda environment does not change the units attribute (OK).

* ESMValCore 2.9 with a v2.9 conda environment changes the units attribute (ERROR)

I could trace back the problem of the broken units attribute in the preprocessed files to the preprocessor "area_statistics " (operator sum). I would therefore expect, all recipes using this preprocessor might also be affected, even though this probably shows in weird labels in the plot only.

Given the tests I did, I would think this is a problem of a changed dependency of the core and should probably be best addressed in the core directly.

cheers @axel-lauer - that's probably cf_units via iris so we can't really do much about it, would you be OK to merge this now, then maybe open an issue in Core and we can go do stuff there? Not to be a partypooper but this is the last M2.9 PR 🤓

[schlunma]

Please do not merge, this can be probably included in iris (SciTools/iris#5341). Will add details once this is done 👍

[remi-kazeroni]

Following this #3228 (comment)

[valeriupredoi]

no probs, but it may need an M2.10 change then unless that gets tossed in iris 3.6.x on time, checks and run happen, and @bouweandela is still in Amsterdam, and not on some beach in the Azores 👍

[schlunma]

Thanks @axel-lauer for your comment, that makes a lot of sense. Indeed this is a new "feature" of iris 3.6 (again introduced by myself 😬) and boils down to Unit('kg m-2 s-1') * Unit('1') = Unit('m-2.kg.s-1').

I now fixed this in iris in a PR that will be included in iris v3.6.1 (SciTools/iris#5341), so this PR can probably be closed. Will re-open if necessary.

[valeriupredoi]

Manu, I'd keep this open and hanging about until that gets into stable iris, if I was you, bud

[schlunma]

The corresponding issue is still open (#3224), so we should be fine 😁

[valeriupredoi]

Sounds good, we need something for potential users coming at us 😁