Update variant nowcast target data to reflect the latest standards and decicions

[bsweger]

Background

In the weeks since we began generating target data for the variant nowcast hub, we've identified two changes to make.

oracle output

Because the oracle output files are not partitioned by sequence_as_of, they will be overwritten each time the hub's post-submission jobs are run (if the round_id in question is still in the 14 week window interim data window).

For normal hub operations, this is fine. However, if the target data jobs are run out of order (for example, a manual backfill run with an older nowcast_date), that would break the implicit assumption that oracle.parquet fill always reflect the most recent nowcast_date.

We decided that partitioning oracle outputs by sequence_date makes them too onerous to access. So we can't guarantee that overwriting the file won't happen, but we can add sequence_as_of information to the oracle output files as a breadcrumb.

time series

The RFC for Hubverse time series target data formats was approved on 2025-02-28 🎉 .

Based on the new time series guidance, we decided to:

• Partition time series target data onas_of, where as_of represents the sequence_as_of date
• Use a secondary partition of nowcast_date
• Explicitly include the above partition fields as columns in the time series parquet files
• Remove tree_as_of from the time series data (there is no immediate use case, and we're able to get the tree_as_of date from the created_at field in themodeled_clades json (for example))

Definition of done

• The oracle.parquet files generated by get_target_data.py will contain an as_of column that represents sequence_as_of
• The time series parquet files are partitioned by as_of (aka sequence_as_of) and then by nowcast_date
• The current time series parquet column sequence_as_of will be renamed to as_of
• The current time series parquet column tree_as_of will be removed
• The hub's target-data README will contain a data dictionary for both time series and oracle output target data formats

[bsweger]

I also added a tree_as_of column as part of this work, even though we didn't explicitly have that as a requirement. It aligns with our approach for the time series outputs and provides clarify for people accessing this file outside of the hub context.

tree_as_of is cancelled!

[bsweger]

Moving this out of "ready for review" status pending finalization of Hubverse target data formats.

[bsweger]

Picking this back up now that the RFC for time series target data formats has been approved

Will update the ticket's "definition of done" to reflect the outcome of today's variant nowcast hub meeting.

[elray1]

I read through the updated main comment just now and confirmed that it looks right to me.