ww3_ounf - improve gridded output data CF compliance

[ukmo-ansaulter]

This issue covers actions needed to improve CF compliance for gridded output data generated by the ww3_ounf netCDF post processor.

@mickaelaccensi , @thesser1 , @JessicaMeixner-NOAA , @aliabdolali - I'm highlighting this now so that you have early sight of what myself and @ukmo-ccbunney will be working on. Please let us know if you disagree with any of the below.

1. Review and revise CF standard names. A large number of the standard names presently assigned in ww3_ounf are not listed in the current CF tables. Our suggested workplan is twofold: first remove non-compliant names from the code and store in a ounfmeta config namelist, this would allow the current default files to still be generated but through the user configuration not as a default; second begin to work through submission of standard names to the CF discussion site - it is probably best to do this in small batches, starting with the most important parameter groups, rather than drown the discussion group with everything

2. Add the use of grid mapping, (optional) coordinate reference systems and (optional) time references via the variable coordinates attribute

3. If appropriate, add use of cell representation attributes (e.g. bounds, cell_methods)

4. Enable the (optional) use of "seconds from" time units for float and long integer types; these options aren't a CF requirement specifically, but reduce the risk of rounding errors occurring when read by client software

5. Update regtests to generate sample files that can be passed through CF compliance checkers

Please comment if any of this is of concern, doesn't make sense, or you feel we've missed something. Description of the current conventions can be found here

[JessicaMeixner-NOAA]

These all sound like reasonable good things to me. One question I have is how does this then let ounf diverge from outf and grib2 output? I've seen a lot of questions/comments about inconsistency in documentation/output, which might be things on NCEP side and nothing to do with code, I haven't dug into that yet.

[thesser1]

For #2, if possible, I would like to see UGRID standards included for unstructured mesh.

[ukmo-ansaulter]

For #2, if possible, I would like to see UGRID standards included for unstructured mesh.

Agreed Ty. We ought to see how this maps to the SMC grid data as well. Note to self - the standards are here

[ukmo-ansaulter]

These all sound like reasonable good things to me. One question I have is how does this then let ounf diverge from outf and grib2 output? I've seen a lot of questions/comments about inconsistency in documentation/output, which might be things on NCEP side and nothing to do with code, I haven't dug into that yet.

I'm not sure that we address that point in this issue @JessicaMeixner-NOAA ? For this one we aren't aiming to add any netCDF output functionality beyond what is already in ww3_ounf (i.e. it already does rotated, cartesian, SMC and unstructured grids + all of the various diagnostic outputs), we are simply trying to bring the metadata descriptions up to scratch.

My impression has always been that the divergence is really in terms of which variables and grid descriptions GRIB2/ww3_outf can deal with, versus netCDF/ww3_ounf? And that is a whole different can of worms, as I don't have a clue (and would personally prefer not to learn) how flexible GRIB2 actually is?

It is probably worth trying to update the documentation to reflect what is available to each processing method though.

[ukmo-ansaulter]

A few more notes on 2. grid_mapping. Suggested way for the code to be developed might be:

In all cases a (coordinate reference system) CRS variable specific to real world lat-lon coordinates can be added optionally, e.g. to identify that the underlying lat-lon values are specified on a sphere, WGS84 coordinates etc..

Regular lat-lon grid; no default grid mapping as coordinates and dimensions are the same thing

Rotated lat-lon; grid mapping and generation of regular lat-lon coordinates generated and associated via the coordinates attribute as a default (e.g. the rotated pole example 5.6 in CF conventions )

Cartesian grid - as per rotated(?)

SMC grid; minimum default option - regular lat-lon coordinates generated and associated via the coordinates attribute; better option - also define bounds variables for cell faces/edges (in SMC grid terminology this is not the same as edge connections for unstructured grids in the UGRID conventions)

Unstructured grid; minimum default option - regular lat-lon coordinates generated and associated via the coordinates attribute; better option - define as 2D triangular mesh topology

Step one in the code development should be to get these minimum requirements all coded in. Then we can revisit the additional work needed for the SMC/unstructured grid specs....

[ukmo-ccbunney]

With respect to (4), one of the things that I changed in the Met Office's version 4.18 of ww3_ounf was to do all the time calculations internally in seconds (stored as an 8 byte integer). This avoided any truncation/rounding errors associated with working in "day fractions" and allowed for easy output of either "seconds" (as an int64) or "days" (as a double) for the time variable. In this way, the time handling is more like that used in UNIX systems, but with a 64bit rather than 32bit integer so the max date is not limited.

Would this be something that we might like to apply to the current codebase?

Edit: Just to clarify above, I mean that all the datetime offsets are stored as seconds

[hot007]

Hi folks, sorry to comment on a closed issue, but seeking some advice please.

We (Bureau of Meteorology and CSIRO Australia) are currently in the process of preparing some SMC grid WW3 data for publication. I'd like to make the data as standards compliant as possible, and to this end we're very thankful for the improved CF compliance of WW3 output :)
However, being technically unstructured grid data, I feel we ought to try to adhere to UGRID in the hope of making the data usable with at least some tools-other-than-python. I wonder if @thesser1 or @ukmo-ansaulter came to any conclusion on how to construct the required edge/face connections UGRID expects, for SMC data? We can try to reverse-engineer something but it'd be good to understand what you were thinking from a developer perspective how this could be done. Is there a way we can parse the grid files supplied to ww3_grid to extract the connectivity?
To be honest, we are seriously considering just not being UGRID compliant for this dataset, and instead providing some demonstration code for how to work with the data alongside the publication. But if you have some guidance we're all ears :)

thanks,
Claire

[ukmo-ccbunney]

Hi folks, sorry to comment on a closed issue, but seeking some advice please.

We (Bureau of Meteorology and CSIRO Australia) are currently in the process of preparing some SMC grid WW3 data for publication. I'd like to make the data as standards compliant as possible, and to this end we're very thankful for the improved CF compliance of WW3 output :) However, being technically unstructured grid data, I feel we ought to try to adhere to UGRID in the hope of making the data usable with at least some tools-other-than-python. I wonder if @thesser1 or @ukmo-ansaulter came to any conclusion on how to construct the required edge/face connections UGRID expects, for SMC data? We can try to reverse-engineer something but it'd be good to understand what you were thinking from a developer perspective how this could be done. Is there a way we can parse the grid files supplied to ww3_grid to extract the connectivity? To be honest, we are seriously considering just not being UGRID compliant for this dataset, and instead providing some demonstration code for how to work with the data alongside the publication. But if you have some guidance we're all ears :)

thanks, Claire

Hello @hot007

Apologies for the tardy reply to this.

I've considered whether it would be worthwhile trying to make the SMC grid outputs UGRID compliant, but I think for the time being there would be a lot of work involved to do so. Ideally we would want to actually generate the grids in something akin the UGRID representation - i.e. with a set of nodes and connections. However, this would require a lot of work from @ukmo-jianguo-li and @ukmo-ansaulter to update the grid generation code and the internal representation in the wave model itself, so I don't think this will be happening any time soon.

There is also the question of whether UGRID representation in the output files is actually the best format for the SMC grid; because it is actually a quasi-rectangular grid, it can actually be stored quite efficiently with a "cell centres and bounds" approach (although admittedly in this case the current output netCDF format could be improved!).

Saying that, I did write a quick bit of Python code that converted an SMC netCDF grid output file to UGRID format - it's not actually too tricky other than having an efficient method to remove all the duplicate node points (of which there will be many if you construct nodes points from each cell corner). The file size is larger for the UGRID format due to the increased complexity of the mesh data, but this can be mitigated slightly if you store many variables in one file.

[hot007]

Thanks @ukmo-ccbunney that pretty much aligns with what we were thinking. If UGRID compliance is added in a future release we'll happily switch to it but for now I'm not sure the benefits outweigh the effort and storage. I'd probably feel different if QGIS leapt to visualise anything that's UGRID compliant but we already know it doesn't so I don't think lack of compliance is worth considering a hindrance at this stage specifically for SMC grid data.