ESGF facets for CORDEX-Adjust, CORDEX-ESD, CORDEX-Reklies

[larsbuntemeyer]

Is your feature request related to a problem? Please describe.
esmvalcore.esgf.find_files will not find any files for CORDEX related projects, e.g.,

• CORDEX-Reklies: data produced in the REKLIES project (not really clear to me why this is separate project in ESGF)
• CORDEX-Adjust: bias adjusted scenarios
• CORDEX-ESD: statistically downscaled datasets

Would you be able to help out?
The solution is straight forward, e.g., simply adding these projects to esmvalcore.esgf.facets.

[larsbuntemeyer]

Note, that search results will depend on the ESGF host for, at least, CORDEX-Adjust, e.g., consider this:

import requests

hosts = [
    "esgf.ceda.ac.uk",
    "esgf-node.llnl.gov",
    "esgf-data.dkrz.de"
]

def request(
    host, project="CORDEX-Adjust", type="Dataset", **search,
):
    print(f"searching: {host}")
    url = f"https://{host}/esg-search/search"
    version = search.get("version", None)
    if type == "File" and version:
        # this does not work for File searches since version denotes here rcm_version
        del search["version"]
    elif version and version.startswith("v"):
        search["version"] = version[1:]
    params = dict(
        project=project,
        type=type,
        format="application/solr+json",
        limit=None,
        distrib=True,
        replica=False,
    )
    params.update(search)
    return requests.get(url, params)


facets = {
    "rcm_name": "REMO2009",
    "variable": "tasAdjust",
}
responses = [request(h, **facets) for h in hosts]
[r.json()["response"]["numFound"] for r in responses]

results in:

searching: esgf.ceda.ac.uk
searching: esgf-node.llnl.gov
searching: esgf-data.dkrz.de

[10, 0, 24]

Did someone encounter similiar cases before? I see that esmvalcore favours esgf.ceda.ac.uk here:

ESMValCore/esmvalcore/config/_esgf_pyclient.py

Lines 145 to 154 in 469fd09

	'urls': [
	'https://esgf.ceda.ac.uk/esg-search',
	'https://esgf-node.llnl.gov/esg-search',
	'https://esgf-data.dkrz.de/esg-search',
	'https://esgf-node.ipsl.upmc.fr/esg-search',
	'https://esg-dn1.nsc.liu.se/esg-search',
	'https://esgf.nci.org.au/esg-search',
	'https://esgf.nccs.nasa.gov/esg-search',
	'https://esgdata.gfdl.noaa.gov/esg-search',
	],

[bouweandela]

The solution is straight forward, e.g., simply adding these projects to esmvalcore.esgf.facets.

A pull request would be most welcome. Usually, the most difficult part is to figure out how to translate the names used on ESGF to the ESMValTool vocabulary.

Did someone encounter similiar cases before?

Yes, I've seen similar issues with CMIP3:

ESMValCore/esmvalcore/config/_esgf_pyclient.py

Line 144 in e3cc51b

# Be careful about the url, not all search urls have CMIP3 data?

Actually, that was quite a poor place to document that behaviour, a better place would have been here.

Related to that: on the [email protected] mailing list it was recently mentioned that there is currently an issue with the CEDA index node:

It seems the CEDA index does not show data published on
esg-dn1.nsc.liu.se. Probably something with the shard replication went wrong. I cc'ed the admins of that node, they should be able to fix this.

Unfortunately I can not recommend a particular index as all of them have
failing shards every now and then.