consider provision of index files in notification message

[yhe-wmo]

As noted in the TT-NWPMD meeting on 07.03.2023, TT-NWPMD will further discuss how to provide index files of GRIB2 (e.g. the key value of the notification message, the format of the index files, and mandatory/optional requirement) in the WIS2 notification message.

[sebvi]

I think index files are very important, especially if we allow data to be published aggregated in different ways. It could help in the extraction of specific subsets

[alexandreleroux]

As discussed at the TT-NWPMD meeting today, given this index file is 'metadata' about a specific GRIB2 file, I wonder if this metadata could actually be included in the corresponding wcmp2 record.

Thus one metadata record (wcmp2) instead of two (wcmp2+index) for a single GRIB2 file?

[yhe-wmo]

TT-NWPMD Meeting 2023.06.13 also noted that, if we are going to recommend the provision of index file, the TT should ensure consistency among data providers on the syntax, data format, etc. of the index file. TT decided to revisit this issue later, meanwhile, will seek clarification from TT-WISMD the possibility of integrating the information of the index file into the WCMP2 record.

[amilan17]

@tomkralidis Please note that I added this issue to the TT-WISMD project board.

[josusky]

Let me ask a stupid question, as is usual for me and as I have very little knowledge of NWP products:
Is the index file static in the sense that the same index file is valid for the GRIB2 product produced yesterday and GRIB2 product produced today?
If the answer is yes, then it can be part of the metadata - possibly as a link, not directly included in the JSON file, as I expect it to be somewhat bigger - but here again, my assumption about the size could be wrong.

[sebvi]

There are no stupid questions :)

The index file is not static in nature. GRIB2 is a record based format which means a GRIB file contains a collection of completely independent GRIB records in it. The index file usually contains useful keys to index the records (that part is essentially static) and also offset octet and octet length of each record to allow fast access/positioning in a file (that part is usually not static due to built in compression of the data values, i.e. the records vary in size from day to day)

HAve a look here for ECMWF index files: ECMWF open data

[tomkralidis]

TT-WISMD 2023-06-22:

• considering https://data.ecmwf.int/forecasts/20230619/00z/0p4-beta/oper/
  • perhaps the data producer can zip both the .grib2 and .index file and emit a message of the zipfile? Yes, this does imply non-trivial infrastructure changes. This would also help with the global cache providing both files
  • can we add a link to all index files as a additional link with rel=related?
  • one can also provide a link to the directory listing with rel=related
• this also introduces issues if/when using the properties.content field on how to provide information on both
• in general, we would like to avoid data specific information as part of WNM
• complete discussion at https://github.com/wmo-im/tt-wismd/wiki/Meeting-2023-06-22#notes
• in general, we need to consider coarser notifications as part of consider coarser resource publication with interactive links wis2-notification-message#41

[amilan17]

@wmo-im/tt-nwpmd meeting on 5 July 2023 were satisfied with the approach of providing a related link to the index file.

[amilan17]

see last comment in this issue: wmo-im/wis2-notification-message#41

[golfvert]

By design, WIS2 is (and MUST stay) data format agnostic. The notification message, the topic hierarchy don't care whether it is a GRIB a NETCDF or else.
I understand the issue is sorted with the agreed mechanism.

To be closed.

[yhonda21]

NWPMD meeting on 2023.09.14

• TT-WISMD agree to include a link to the index file in the notification message. The index file itself won't be cached. TT-NPWMD won't define the format of the index file and agreed that each Centre can use its own format of the index file. The ticket consider provision of index files in notification message #13 is closed.