Meeting 2023 11 06

Date/Time/Location

Monday 6 November 2023, 1200-1400 UTC, MS Teams

Proposed Agenda

• Discussion with Enrico [60 min]
• Reviewing issues on GitHub repository [20 min]​
• Review of the past meetings [10 min]​
• Progress on ECMWF's WIS2.0 Pilot Phase [10 min] ​
• NWP metadata and WCMP2 [50 min]​
• Next meeting and wrap up [10 min]​

see ppt of the meeting : (https://wmoomm.sharepoint.com/:p:/s/wmocpdb/ERLr-SurhCtMvfn8U0xU1R0BnqtuSnL4K7PcUfeXORGt7w?e=JCrbHF)

List of participants

Name	Agency	Attendance
Mr Sébastien VILLAUME (Chair)	ECMWF	X
Mr Sebastian BORCHERT	DWD	X
Mr Thomas COLEMAN	BoM
Mr Masato FUJIMOTO	JMA
Mr Mark HEDLEY	UKMO
Mr Arun KUMAR	NOAA
Mr Alexandre LEROUX	MSC	X
Ms Yuanyuan LIU	CMA	X
Mr Vijay TALLAPRAGADA	NOAA
Mr Andrew Benjamin	NOAA	X
Mr Radomir ZARIPOV	ROSHYDROMET
Mr Enrico FUCILE	Secretariat	X
Ms Anna MILAN	Secretariat	X
Mr Hassan HADDOUCH	Secretariat
Mr Yuki HONDA	Secretariat	X
Ms Eunha LIM	Secretariat
Ms Sohee PARK	Secretariat	X

Notes

1. Discussion with Enrico

TT-NWPMD discussed #12 mainly.

• TT-WISMD is discussing this issue. The current idea on TH doesn't seem flexisible enough. NWP Centres generate model output files in different ways. If no flexibility, it won't work. This issue should be further discussed by TT-WISMD at its next in-person meeting.
• The extra level of details could be described in the discovery metadata. But the flexisibility should be also controled.
• The current proposed TH has 13 levels. But the country and center levels are merged into one level at the higher level. So the levels of TH will be 12.
• Some centres generate core and recommended data in one file and put a part of core data in another file under other TH. From the monitoring purpose, all core data should be in one file.
• WIS already gave the guidance to us to split core and recommended data into different files. But it seems that this guidance is not confirmed yet by both WIS and WIPPS.
  • Even if a Centre send a file including both core and recommended data to Global Cache, WIS won't care if the contained variables are more than those defined in the Manual on WIPPS. But note that all such data will be shared on a free and unristricted basis.
• Tropical cyclone variables are point data. If there is one more TH, users cannot recognized NWP grid outputs and TC point outputs. An extra level will be needed for TH from the WIS perspective. But it looks unrealistic to create THP for every potential types of products. At the same time, WIS won't standardize the file name, so the file name cannot be used to replace the role of extra level of TH. A topic for TC variables would be necessary to represent them in the metadata catalog.
• Another thing is the notification messsages where you may be able to add metadata ID. Then users will receive the notification and filter the data using metadata ID in the notification messages. But this may not be effieient since clients need to write some Python scripts to filter. It is therefore recommended to add one more level of TH rather than filtering using metadata ID. Freedom to add more levels of TH which are not standardized is important.
• The links to files can be included in the notification message while the no standardization of file names. MQTT notification messages should allow us to filter the data. If so, this idea should be proposed to WIS architecture team. Currently you can filter the data only on geographical domains. If you prefer to filter more options, the notification messages should be standardized more. We should choose either extra level of TH or filering option of notification message to filter data type of products.
• TH won't be included in the notification message, but it will be in the discovery metadata record.
• Expected use of WIS: Users search data using metadata catalog and find the TH for that data. Users subscribe TH and receive the notification when the data is published.
  • If many data are associated with one TH, users may receive a lot of notifications for data most of which users don't use.
• The meeting recognize the need to understand how notification messages, metadata IDs and TH are related.

2. Reviewing issues on GitHub repository

Updates of TT-NWPMD issues

• #12:
  • Filtering using notification messages is limited only for geographical domains. Although the architecture team cosidered this ticket could be closed, this ticket should be kept open to further discuss TH including the method to filter data.
• #14: no progress.
• #16: This ticket will be revisited when the pilot phase is over.
• #17:
  • Alex reported the progress at ECCC (see ticket #17). The produced metadata are not well examined yet.
  • Sebastien pointed out some issue. ID shows the data is humidity, but it doens' give the information the level (e.g. 700hPa). You need to search key words in the title. The discovery metadata seems quite similar for those of 700hPa's and 800hPa's humidity. This is the reason why we need to define the metadata properly. But WCMP2 may not have enough key words, for example to describe the vertical levels.
  • In the case of ECMWF, all variables are included in one file. But it doesn't mean all variables are defined in all vertical levels. Sebastien explained the ECMWF's discovery metadata that describe a set of parameter lists for each contained variable.
  • Alex opined that the link of MQTT notification message to the corresponding WCMP2 metadata record is important for users to filter out based on keywords.
  • Anna requested Sebastien to create a ticket regarding "concept" and keywords for the discovery metadata.

Updates from WIS colleagues

• #44: WIS architechture team will discuss this next week. As Enrico says, TH should be more flexible. The approval process to update TH is still discussed.

(Other agenda items were skipped due to the shortage of the time.)

3. Next meeting and wrap up

• The next meeting will be scheduled on 4 December 2023, 1200-1400 UTC, MS Teams