Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Scaling up the recipe with damip request #8

Merged
merged 6 commits into from
Jun 16, 2022
Merged

Scaling up the recipe with damip request #8

merged 6 commits into from
Jun 16, 2022

Conversation

jbusecke
Copy link
Collaborator

This PR implements the first query from #3. I used my WIP parsing logic locally to generate the list of dataset_ids in the recipe and ran some quick QC locally:

  • There are two datasets (commented out) for which the ESGF API does not provide any urls. We need a good way to handle these cases (since they might be possible just intermittent). Ideas welcome.

  • There is an issue with some of the datasets being provided in fairly small files. @cisaacstern and I have talked about this and this can probably be resolved by expanding the dynamic kwargs logic to not just subset files but also combine them with inputs_per_chunk being determined dynamically aswell.

But for now lets try these and see if that helps @rebeccaherman1

@pangeo-forge-bot
Copy link
Collaborator

🎉 New recipe runs created for the following recipes at sha 187df16a2b3263e2bb77f3d7c98042e4d6cc545e:

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=283

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=183

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/283

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/183

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (bnds: 2, lat: 180, lon: 288, time: 2040)
    Coordinates:
      * bnds       (bnds) float64 1.0 2.0
      * lat        (lat) float64 -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
      * lon        (lon) float64 0.625 1.875 3.125 4.375 ... 355.6 356.9 358.1 359.4
      * time       (time) object 1850-01-16 12:00:00 ... 2019-12-16 12:00:00
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(180, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(288, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(120, 180, 288), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(120, 2), meta=np.ndarray>
    Attributes: (12/46)
        Conventions:            CF-1.7 CMIP-6.0 UGRID-1.0
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  36500.0
        comment:                <null ref>
        ...                     ...
        table_id:               Amon
        title:                  NOAA GFDL GFDL-ESM4 model output prepared for CMI...
        tracking_id:            hdl:21.14100/bec86553-14a4-4b86-874c-552065a85b2b
        variable_id:            pr
        variant_info:           N/A
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-283/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@pangeo-forge-bot
Copy link
Collaborator

Pangeo Forge Cloud told me that our test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 failed. But don't worry, I'm sure we can fix this!

To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/183

If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.

Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r3i1p1f1.Amon.pr.gn.v20190508 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/185

@pangeo-forge-bot
Copy link
Collaborator

When I tried to import your recipe module, I encountered this error

            line 102, in esgf_search
        r = client.get(url)
      File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='esgf-node.llnl.gov', port=443): Max retries exceeded with url: /esg-search/search/?activity_id=DAMIP&institution_id=MIROC&source_id=MIROC6&experiment_id=hist-aer&member_id=r1i1p1f1&table_id=Amon&variable_id=pr&grid_label=gn&project=CMIP6&type=File&distrib=false&format=application%2Fsolr%2Bjson&limit=500&offset=0 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f014a5d6df0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Please correct your recipe module so that it's importable.

@pangeo-forge-bot
Copy link
Collaborator

Pangeo Forge Cloud told me that our test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r3i1p1f1.Amon.pr.gn.v20190508 failed. But don't worry, I'm sure we can fix this!

To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/185

If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.

Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!

@jbusecke jbusecke mentioned this pull request May 25, 2022
Closed
@jbusecke
Copy link
Collaborator Author

jbusecke commented Jun 1, 2022

Ok I think I have figured the netcdf detection out! Thanks to @rabernat @cisaacstern and this doc.

I have reduced the number of datasets to one member per model (assuming that the file format for the output does not change between modelling centers...

cisaacstern
cisaacstern reviewed Jun 1, 2022
View reviewed changes
feedstock/recipe.py
@@ -330,6 +330,12 @@ def choose_chunksize(
else:
return output_chunksizes[0]

def is_netcdf3(url:str) -> bool:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@pangeo-forge-bot
Copy link
Collaborator

🎉 New recipe runs created for the following recipes at sha fd5319b4bbc754765be02ecedf632b0b8518da3f:

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=323

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/323

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 160, bnds: 2, lon: 320, time: 2052)
    Coordinates:
      * lat        (lat) float64 -89.14 -88.03 -86.91 -85.79 ... 86.91 88.03 89.14
      * lon        (lon) float64 0.0 1.125 2.25 3.375 ... 355.5 356.6 357.8 358.9
      * time       (time) object 1850-01-16 12:00:00 ... 2020-12-16 12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(160, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(320, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(684, 160, 320), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(684, 2), meta=np.ndarray>
    Attributes: (12/49)
        Conventions:            CF-1.7 CMIP-6.2
        activity_id:            DAMIP
        branch_method:          Standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  2289.0
        cmor_version:           3.3.2
        ...                     ...
        table_id:               Amon
        table_info:             Creation Date:(30 July 2018) MD5:e53ff52009d0b97d...
        title:                  BCC-CSM2-MR output prepared for CMIP6
        tracking_id:            hdl:21.14100/5b87ebf6-35d8-499f-8c2a-b6814243ff35
        variable_id:            pr
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-323/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=324

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/324

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 64, bnds: 2, lon: 128, time: 2052)
    Coordinates:
      * lat        (lat) float64 -87.86 -85.1 -82.31 -79.53 ... 82.31 85.1 87.86
      * lon        (lon) float64 0.0 2.812 5.625 8.438 ... 348.8 351.6 354.4 357.2
      * time       (time) object 1850-01-16 12:00:00 ... 2020-12-16 12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(64, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(128, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(2052, 64, 128), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(2052, 2), meta=np.ndarray>
    Attributes: (12/53)
        CCCma_model_hash:            55f484f90aff0e32c5a8e92a42c6b9ae7ffe6224
        CCCma_parent_runid:          rc3.1-pictrl
        CCCma_pycmor_hash:           33c30511acc319a98240633965a04ca99c26427e
        CCCma_runid:                 rc3.1-aer10
        Conventions:                 CF-1.7 CMIP-6.2
        YMDH_branch_time_in_child:   1850:01:01:00
        ...                          ...
        table_info:                  Creation Date:(20 February 2019) MD5:374fbe5...
        title:                       CanESM5 output prepared for CMIP6
        tracking_id:                 hdl:21.14100/2e1b64af-6522-4300-b484-6e88c72...
        variable_id:                 pr
        variant_label:               r10i1p1f1
        version:                     v20190429

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-324/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=325

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r10i1p1f2.Amon.pr.gr.v20190308 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/325

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r10i1p1f2.Amon.pr.gr.v20190308 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:      (lat: 128, lon: 256, time: 2052, axis_nbounds: 2)
    Coordinates:
      * lat          (lat) float64 -88.93 -87.54 -86.14 -84.74 ... 86.14 87.54 88.93
      * lon          (lon) float64 0.0 1.406 2.812 4.219 ... 354.4 355.8 357.2 358.6
      * time         (time) datetime64[ns] 1850-01-16T12:00:00 ... 2020-12-16T12:...
    Dimensions without coordinates: axis_nbounds
    Data variables:
        pr           (time, lat, lon) float32 dask.array<chunksize=(1026, 128, 256), meta=np.ndarray>
        time_bounds  (time, axis_nbounds) datetime64[ns] dask.array<chunksize=(1026, 2), meta=np.ndarray>
    Attributes: (12/52)
        CMIP6_CV_version:       cv=6.2.3.0-7-g2019642
        Conventions:            CF-1.7 CMIP-6.2
        EXPID:                  CNRM-CM6-1_hist-aer_r10i1p1f2
        activity_id:            DAMIP
        arpege_minor_version:   6.3.2
        branch_method:          standard
        ...                     ...
        table_id:               Amon
        title:                  CNRM-CM6-1 model output prepared for CMIP6 / DAMI...
        tracking_id:            hdl:21.14100/041c0ce4-08a6-4346-9e75-3d1954cfbadd
        variable_id:            pr
        variant_label:          r10i1p1f2
        xios_commit:            1442-shuffle

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-325/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r10i1p1f2.Amon.pr.gr.v20190308.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=326

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20201120 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/326

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20201120 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 144, bnds: 2, lon: 192, time: 2052)
    Coordinates:
      * lat        (lat) float64 -89.38 -88.12 -86.88 -85.62 ... 86.88 88.12 89.38
      * lon        (lon) float64 0.9375 2.812 4.688 6.562 ... 355.3 357.2 359.1
      * time       (time) datetime64[ns] 1850-01-16T12:00:00 ... 2020-12-16T12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(144, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(192, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(1026, 144, 192), meta=np.ndarray>
        time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(1026, 2), meta=np.ndarray>
    Attributes: (12/47)
        Conventions:            CF-1.7 CMIP-6.2
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  0.0
        cmor_version:           3.4.0
        ...                     ...
        table_info:             Creation Date:(30 April 2019) MD5:9328082e1e6d1da...
        title:                  ACCESS-CM2 output prepared for CMIP6
        tracking_id:            hdl:21.14100/0fe7e095-a3e5-4064-9790-f508c65a586f
        variable_id:            pr
        variant_label:          r1i1p1f1
        version:                v20201120

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-326/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20201120.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=327

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=332

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p1f1.Amon.pr.gn.v20180821 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/332

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p1f1.Amon.pr.gn.v20180821 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 90, bnds: 2, lon: 144, time: 1212)
    Coordinates:
      * lat        (lat) float64 -89.0 -87.0 -85.0 -83.0 ... 83.0 85.0 87.0 89.0
      * lon        (lon) float64 1.25 3.75 6.25 8.75 ... 351.2 353.8 356.2 358.8
      * time       (time) object 1850-01-16 12:00:00 ... 1950-12-16 12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(90, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(144, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(12, 90, 144), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(12, 2), meta=np.ndarray>
    Attributes: (12/48)
        Conventions:            CF-1.7 CMIP-6.2
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  0.0
        cmor_version:           3.3.2
        ...                     ...
        table_id:               Amon
        table_info:             Creation Date:(21 March 2018) MD5:652eaa766045a77...
        title:                  GISS-E2-1-G output prepared for CMIP6
        tracking_id:            hdl:21.14100/222f4082-4de6-41f2-8a39-2441d5187b83
        variable_id:            pr
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-332/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p1f1.Amon.pr.gn.v20180821.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=333

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=334

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NCAR.CESM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20200206 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/333

@cisaacstern
Copy link
Member

/run recipe-test recipe_run_id=335

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r1i1p1f1.Amon.pr.gn.v20190920 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/334

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/335

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NCAR.CESM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20200206 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 192, nbnd: 2, lon: 288, time: 1200)
    Coordinates:
      * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
      * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
      * time       (time) object 1850-01-15 12:00:00 ... 1949-12-15 12:00:00
    Dimensions without coordinates: nbnd
    Data variables:
        lat_bnds   (lat, nbnd) float64 dask.array<chunksize=(192, 2), meta=np.ndarray>
        lon_bnds   (lon, nbnd) float64 dask.array<chunksize=(288, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(600, 192, 288), meta=np.ndarray>
        time_bnds  (time, nbnd) object dask.array<chunksize=(600, 2), meta=np.ndarray>
    Attributes: (12/45)
        Conventions:            CF-1.7 CMIP-6.2
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   674885.0
        branch_time_in_parent:  306600.0
        case_id:                1635
        ...                     ...
        sub_experiment_id:      none
        table_id:               Amon
        tracking_id:            hdl:21.14100/2dc00c6d-f907-4401-87e3-b5d144ad0f5a
        variable_id:            pr
        variant_info:           Historical aerosol-only simulations resemble the ...
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-333/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NCAR.CESM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20200206.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r1i1p1f1.Amon.pr.gn.v20190920 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 96, bnds: 2, lon: 144, time: 240)
    Coordinates:
      * lat        (lat) float64 -90.0 -88.11 -86.21 -84.32 ... 86.21 88.11 90.0
      * lon        (lon) float64 0.0 2.5 5.0 7.5 10.0 ... 350.0 352.5 355.0 357.5
      * time       (time) object 1850-01-16 12:00:00 ... 1869-12-16 12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(96, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(144, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(60, 96, 144), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(60, 2), meta=np.ndarray>
    Attributes: (12/49)
        Conventions:               CF-1.7 CMIP-6.2
        activity_id:               DAMIP
        branch_method:             Hybrid-restart from year 1600-01-01 of piControl
        branch_time:               0.0
        branch_time_in_child:      0.0
        branch_time_in_parent:     430335.0
        ...                        ...
        table_id:                  Amon
        table_info:                Creation Date:(24 July 2019) MD5:0bb394a356ef9...
        title:                     NorESM2-LM output prepared for CMIP6
        tracking_id:               hdl:21.14100/bf65aa38-460f-4c17-a971-6fc3a234caf0
        variable_id:               pr
        variant_label:             r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-334/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r1i1p1f1.Amon.pr.gn.v20190920.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (bnds: 2, lat: 180, lon: 288, time: 2040)
    Coordinates:
      * bnds       (bnds) float64 1.0 2.0
      * lat        (lat) float64 -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
      * lon        (lon) float64 0.625 1.875 3.125 4.375 ... 355.6 356.9 358.1 359.4
      * time       (time) object 1850-01-16 12:00:00 ... 2019-12-16 12:00:00
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(180, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(288, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(120, 180, 288), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(120, 2), meta=np.ndarray>
    Attributes: (12/46)
        Conventions:            CF-1.7 CMIP-6.0 UGRID-1.0
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  36500.0
        comment:                <null ref>
        ...                     ...
        table_id:               Amon
        title:                  NOAA GFDL GFDL-ESM4 model output prepared for CMI...
        tracking_id:            hdl:21.14100/bec86553-14a4-4b86-874c-552065a85b2b
        variable_id:            pr
        variant_info:           N/A
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-335/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@jbusecke
Copy link
Collaborator Author

jbusecke commented Jun 2, 2022

Yay, this looks like it works huh? Should I uncomment the remaining instance ids?

@cisaacstern
Copy link
Member

Should I uncomment the remaining instance ids?

Just to keep this intelligible, could we merge this, and then open a new PR which uncomments the remaining IDs?

@pangeo-forge-bot
Copy link
Collaborator

🎉 New recipe runs created for the following recipes at sha 7c9dec327daae781106aeb689467fb5b9a1dc8d3:

@jbusecke
Copy link
Collaborator Author

/run recipe-test recipe_run_id=360

@jbusecke
Copy link
Collaborator Author

/run recipe-test recipe_run_id=361

@pangeo-forge-bot
Copy link
Collaborator

🎉 New recipe runs created for the following recipes at sha 5df52c403e529f5b469b3550d42de4bfb1c46ed7:

@jbusecke
Copy link
Collaborator Author

/run recipe-test recipe_run_id=362

@jbusecke
Copy link
Collaborator Author

/run recipe-test recipe_run_id=363

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/363

@pangeo-forge-bot
Copy link
Collaborator

✨ A test of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 is now running on Pangeo Forge Cloud!

I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)

In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/362

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (lat: 160, bnds: 2, lon: 320, time: 2052)
    Coordinates:
      * lat        (lat) float64 -89.14 -88.03 -86.91 -85.79 ... 86.91 88.03 89.14
      * lon        (lon) float64 0.0 1.125 2.25 3.375 ... 355.5 356.6 357.8 358.9
      * time       (time) object 1850-01-16 12:00:00 ... 2020-12-16 12:00:00
    Dimensions without coordinates: bnds
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(160, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(320, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(684, 160, 320), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(684, 2), meta=np.ndarray>
    Attributes: (12/49)
        Conventions:            CF-1.7 CMIP-6.2
        activity_id:            DAMIP
        branch_method:          Standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  2289.0
        cmor_version:           3.3.2
        ...                     ...
        table_id:               Amon
        table_info:             Creation Date:(30 July 2018) MD5:e53ff52009d0b97d...
        title:                  BCC-CSM2-MR output prepared for CMIP6
        tracking_id:            hdl:21.14100/5b87ebf6-35d8-499f-8c2a-b6814243ff35
        variable_id:            pr
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-362/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@pangeo-forge-bot
Copy link
Collaborator

🥳 Hooray! The test execution of your recipe CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701 succeeded.

Here is a static representation of the dataset built by this recipe:

            <xarray.Dataset>
    Dimensions:    (bnds: 2, lat: 180, lon: 288, time: 2040)
    Coordinates:
      * bnds       (bnds) float64 1.0 2.0
      * lat        (lat) float64 -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
      * lon        (lon) float64 0.625 1.875 3.125 4.375 ... 355.6 356.9 358.1 359.4
      * time       (time) object 1850-01-16 12:00:00 ... 2019-12-16 12:00:00
    Data variables:
        lat_bnds   (lat, bnds) float64 dask.array<chunksize=(180, 2), meta=np.ndarray>
        lon_bnds   (lon, bnds) float64 dask.array<chunksize=(288, 2), meta=np.ndarray>
        pr         (time, lat, lon) float32 dask.array<chunksize=(120, 180, 288), meta=np.ndarray>
        time_bnds  (time, bnds) object dask.array<chunksize=(120, 2), meta=np.ndarray>
    Attributes: (12/46)
        Conventions:            CF-1.7 CMIP-6.0 UGRID-1.0
        activity_id:            DAMIP
        branch_method:          standard
        branch_time_in_child:   0.0
        branch_time_in_parent:  36500.0
        comment:                <null ref>
        ...                     ...
        table_id:               Amon
        title:                  NOAA GFDL GFDL-ESM4 model output prepared for CMI...
        tracking_id:            hdl:21.14100/bec86553-14a4-4b86-874c-552065a85b2b
        variable_id:            pr
        variant_info:           N/A
        variant_label:          r1i1p1f1

You can also open this dataset by running the following Python code

import fsspec
import xarray as xr

dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-363/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds

in this badge (or your Python interpreter of choice).

Checklist

Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.

Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g, "time" or equivalent) to be 2.

- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?

@jbusecke
Copy link
Collaborator Author

Yayyyy the netcdf3 detection works! Ill merge this now and try to push further on this throughout the day.

@jbusecke jbusecke merged commit 3e8f852 into pangeo-forge:main Jun 16, 2022
@jbusecke jbusecke deleted the first_request_damip branch June 16, 2022 17:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@cisaacstern cisaacstern cisaacstern left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jbusecke @pangeo-forge-bot @cisaacstern