Make derivation of total column ozone (toz) more flexible and add derivation of stratospheric and tropospheric column ozone

[schlunma]

Description

This PR makes the derivation of total column ozone (toz) more flexible by allowing its derivation from

• data that has hybrid levels (e.g., o3 from the AERmon table)
• zonal mean data (e.g., o3 from the AERmonZ table)

In addition, this PR adds two derived variables:

• stratospheric column ozone (soz), where the "stratosphere" is defined as the region where the O3 mole fraction exceeds 125 ppb
• tropospheric column ozone (troz), where the "stratosphere" is defined as the region where the O3 mole fraction is lower than 125 ppb

Backwards-incompatible change

This PR also changes the units of toz in our custom table from DU to m. This is done to make it consistent with the CMIP6 definition. Otherwise, this will lead to various problems when directly comparing CMIP6 output with CMIP5 output (or another project that uses the custom table).

To restore the old behavior, a preprocessor can be added in the recipe to convert m to DU:

convert_units:
  units: DU

---

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible (see above)
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.83%. Comparing base (f7ab83f) to head (5615c74).
Report is 62 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2509      +/-   ##
==========================================
+ Coverage   94.77%   94.83%   +0.05%     
==========================================
  Files         249      251       +2     
  Lines       14095    14191      +96     
==========================================
+ Hits        13359    13458      +99     
+ Misses        736      733       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[schlunma]

This can be tested with the following recipe:

# ESMValTool
---
documentation:
  title: test
  description: test
  authors:
    - schlund_manuel
  maintainer:
    - schlund_manuel

preprocessors:
  du:
    convert_units:
      units: DU

diagnostics:

  test:
    variables:
      toz:
        derive: true
        force_derivation: true
        exp: historical
        start_year: 2013
        end_year: 2014
        preprocessor: du
        additional_datasets:
          - {project: CMIP6, dataset: CESM2-WACCM, ensemble: 'r1i1p1f1', grid: gn, mip: Amon}
          - {project: CMIP6, dataset: GISS-E2-1-G, ensemble: 'r1i1p3f1', grid: gn, mip: AERmon}
      troz:
        derive: true
        force_derivation: true
        exp: historical
        start_year: 2013
        end_year: 2014
        preprocessor: du
        additional_datasets:
          - {project: CMIP6, dataset: CESM2-WACCM, ensemble: 'r1i1p1f1', grid: gn, mip: Amon}
          - {project: CMIP6, dataset: GISS-E2-1-G, ensemble: 'r1i1p3f1', grid: gn, mip: AERmon}
      soz:
        derive: true
        force_derivation: true
        exp: historical
        start_year: 2013
        end_year: 2014
        preprocessor: du
        additional_datasets:
          - {project: CMIP6, dataset: CESM2-WACCM, ensemble: 'r1i1p1f1', grid: gn, mip: Amon}
          - {project: CMIP6, dataset: GISS-E2-1-G, ensemble: 'r1i1p3f1', grid: gn, mip: AERmon}
    scripts:
      null

recipe_test_ozone.yml.txt

[schlunma]

As far as I can tell with, toz and soz look good. However, I get some ugly results for troz, which include lots of artifacts:

Most likely this is the result of lower pressure levels being masked in the input data (o3):

[valeriupredoi]

Manu, can you name these islands?

[schlunma]

Manu, can you name these islands? ![islands](https://private-user-images.githubusercontent.com/28983971

Not all of them, but Google Maps certainly can 😄

[FranziskaWinterstein]

Hey @schlunma, thank you for taking care of this.

I made my own attempt to build troz.py (see below), by basically calculating troz as troz = toz - soz. This reduces the doubling of code, but introduces dependencies from troz on soz. What do you think about that?

"""Derivation of variable ``soz``."""

import dask.array as da
import iris

from ._baseclass import DerivedVariableBase
from .toz import DerivedVariable as Toz
from .soz import DerivedVariable as Soz

class DerivedVariable(DerivedVariableBase):
    """Derivation of variable ``troz``."""

    @staticmethod
    def required(project):
        """Declare the variables needed for derivation."""
        if project == 'CMIP6':
            required = [{'short_name': 'o3'}, {'short_name': 'ps'}]
        else:
            required = [{'short_name': 'tro3'}, {'short_name': 'ps'}]
        return required

    @staticmethod
    def calculate(cubes):
        """Compute tropspheric column ozone.

        Note
        ----
        In the calculation of ``troz``, the surface air pressure (``ps``) is
        used to determine the pressure level width of the lowest layer.

        The calculation of ``troz`` consists of three steps:
        (1) Use derivation function of ``toz`` to calculate ``toz``.
        (2) Use derivation function of ``soz`` to calculate ``soz`` (using the
            masked data).
        (3) Calculate difference of ``toz`` and ``soz`` to calculate ``troz``.

        """
        o3_cube = cubes.extract_cube(
            iris.Constraint(name='mole_fraction_of_ozone_in_air'))
        ps_cube = cubes.extract_cube(
            iris.Constraint(name='surface_air_pressure'))

        # (1) Use derivation function of toz to calculate toz
        cubes = iris.cube.CubeList([o3_cube, ps_cube])
        cube_toz = Toz.calculate(cubes)

        # (2) Use derivation function of soz to calculate soz
        cube_soz = Soz.calculate(cubes)

        # (3) Calculate troz as difference of toz and soz
        return cube_toz - cube_soz

[FranziskaWinterstein]

I successfully tested the given recipe as well as applied it to the diagnostics to plot hovmoeller_vs_lat_or_lon and map plots from multi_dataset.py. No problems.

[FranziskaWinterstein]

The map plots of the climatology 2013 - 2014 of troz do not show those artefacts. I suppose they only appear without time averaging. If it is data dependent, I guess, there is not much we can do about it?

Created with:
recipe_test_ozone_map.yml.txt

[FranziskaWinterstein]

Thank you for sharing this optional of ozone variables. I am impressed by the comprehensive testing.

[schlunma]

I made my own attempt to build troz.py (see below), by basically calculating troz as troz = toz - soz. This reduces the doubling of code, but introduces dependencies from troz on soz. What do you think about that?

I first had it that way and it produced exactly the same results. However, you trade simpler code for speed in this case, as this would mean that the (potential) interpolation needs to be done twice, which can be very expensive. I think avoiding this is better than saving ~20 lines of code.

The map plots of the climatology 2013 - 2014 of troz do not show those artefacts. I suppose they only appear without time averaging. If it is data dependent, I guess, there is not much we can do about it?

Ah, nice to see that time averaging helps! I also think that there's nothing we can really do about it...

Thanks for reviewing 🎉

[schlunma]

@valeriupredoi since you already commented on this PR, would you do us the honor of performing a technical review? Thanks 🍻

[valeriupredoi]

@valeriupredoi since you already commented on this PR, would you do us the honor of performing a technical review? Thanks 🍻

on it right now, bud 🍺

[valeriupredoi]

this looks very good, many thanks! My only (more) serious concern is with those plev points that should be loaded straight from the CMOR table, but all else looks good, cheers 🍺

[valeriupredoi]

Many thanks for addressing the review comments, Manu 🍺