Workshop infrastructure

[KoenGreuell]

The infrastructure requirements for the workshop are known and the necessary infrastructure has been provided:

• The amount of processors the virtual lab uses is known.
• The random access memory and permanent storage usage of the virtual lab are known.
• The number of people taking part in a workshop.
• Run the current workflow to measure its resource usage

[KoenGreuell]

Current plan: Once monitoring works, try to run the workflow using parameters provided by Berend.

[BerendWijers]

Info

1. If you are starting a new dataset, process the PVOL + VP first and visualize after.
2. Not provided here but required for the run
   1. minio endpoint
   2. minio access_key
   3. minio secret_key
   4. knmi api
   5. OPERA_RADAR_DB.json
   6. workflow files
      1. pvol -> vp
      2. visualize ppi
      3. visualize vpts

Processing: Polar Volume (PVOL) + Vertical Profile (VP)

Scenario	radar	start_date	end_date	param_interval
Bad Weather	HRW	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	15
Bad Weather	DHL	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	15
Perfect Migration	HRW	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	15
Perfect Migration	DHL	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	15
Great Weather	HRW	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	15
Great Weather	DHL	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	15

Visualization: Vertical Profile Time Series (VPTS)

Scenario	radar	start_date	end_date
Bad Weather	HRW	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00
Bad Weather	HRW	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00
Bad Weather	DHL	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00
Bad Weather	DHL	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00
Perfect Migration	HRW	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00
Perfect Migration	HRW	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00
Perfect Migration	DHL	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00
Perfect Migration	DHL	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00
Great Weather	HRW	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00
Great Weather	HRW	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00
Great Weather	DHL	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00
Great Weather	DHL	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00

Visualization: Plan Position Indicators (PPI)

Scenario	radar	start_date	end_date	param_param	param_interval	param_elevation
Bad Weather	HRW	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	VRADH	15	0.3
Bad Weather	HRW	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	DBZH	15	0.3
Bad Weather	DHL	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	VRADH	15	0.3
Bad Weather	DHL	2024-07-12T00:00+00:00	2024-07-12T23:55+00:00	DBZH	15	0.3
Perfect Migration	HRW	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	VRADH	15	0.3
Perfect Migration	HRW	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	DBZH	15	0.3
Perfect Migration	DHL	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	VRADH	15	0.3
Perfect Migration	DHL	2024-10-02T18:00+00:00	2024-10-03T12:00+00:00	DBZH	15	0.3
Great Weather	HRW	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	VRADH	15	0.3
Great Weather	HRW	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	DBZH	15	0.3
Great Weather	DHL	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	VRADH	15	0.3
Great Weather	DHL	2025-02-02T00:00+00:00	2025-02-02T23:55+00:00	DBZH	15	0.3

[BerendWijers]

I think we should add OPERA_RADAR_DB.json and the workflow files for the workflows mentioned above to the repository. Then someone only requires to specify their own endpoint, credentials and get their own api key. What do you think @KoenGreuell ?

[KoenGreuell]

Let's try. Can you indeed add:
OPERA_RADAR_DB.json to the codebase directory?
And move the workflows to the workflows directory?
And push all to git? I think by now we can assume that the prepare_lab_for_NAEM branch contains fucntionality we want to keep and it should be merged to main?

[BerendWijers]

Yeah - I'll add the files and indeed I agree we want to keep this functionality so we should merge back to main.

Should we merge after I've added the relevant files?

[KoenGreuell]

I recommend merging first as the smaller the difference between the branches, the easier the merge.

[KoenGreuell]

@BerendWijers I am going to investigate what LifeWatch VLICs experience is with multiple people workshops and multiple people using NaaVRE at the same time right?

[BerendWijers]

Yes, that would be good. We could re-use experience if there is any - if not we'll draft an initial plan.

[KoenGreuell]

Now I remember discussing particularly how data is shared among multiple users of a virtual lab. I will figure out a plan for that in the Make data usable issue: #38

[KoenGreuell]

Sharing data during a workshop

With regards to multiple users during a workshop: we have prior experience in setting up Minio to allow multiple users to read and write from the same repository. This could allow the notebook to check whether some data is already available before downloading or processing the data.