CIL/Rhodium data requests

[cisaacstern]

Here's the list of requested instance ids from @rfofrich, initially provided in pangeo-data/pangeo-cmip6-cloud#38 (comment):

https://gist.github.com/cisaacstern/847f86deaf763e3b4aa6cd062b09d99c

We'll start working on these as soon as the prototyped submission framework looks workable.

[rfofrich]

Sounds great! Our ideal timeline would be to have this completed within the month. However, I completely understand the limitations given you two may be spread thin between projects. Therefore, we do have a few workaround options if you feel that this timeline might be too ambitious.

[cisaacstern]

Thanks for this timeline information, @rfofrich.

@jbusecke and I are working on this repo this week. Once we have a bit more headway (perhaps late next week), I can circle back to this thread with a better estimate on timing for these datasets. If we don't hit too many snags, it's conceivable we could make this work within your stated timeline.

[rfofrich]

@cisaacstern Thank you for the update. Our timeline is somewhat flexible. Therefore, no pressure if it takes slightly longer than expected and I'm looking forward to seeing this further developed. Thank you and @jbusecke again for working on this, it's extremely helpful.

[jbusecke]

Just checking in on this. I think we have made some good progress on the feedstock, but your request might pose some unique challenges since it is daily (compared to the request in #3).
I will spend some time today working further on the logic to dynamically generate kwargs based on filesizes, and could try it out with one of your daily datasets?

[rfofrich]

@jbusecke Thank you for the update and for working on this. Do you have access to the datasets you need to continue?

[jbusecke]

I would get them directly from ESGF if everything works according to plan. So I should be good on that side. Ill keep you posted on the progress.

[jbusecke]

Apologies for the long wait here, but we have made significant progress on ingesting new stores via apache-beam at a larger scale recently.
I am consolidating all the work in cmip6-leap-feedstock and will archive this repo soon.
If you are still interested in the requested data, please resubmit the request here.