Weird Weather Model Variable t Missing in the Dataset Error: ERA5 issue [BUG]

[mohseniaref]

Describe the bug

After installing RAiDER version 0.5.3 ,Python3.10 using Conda on Ubuntu 24, the following error occurs when running the weather model processing. After spending some time, It is ERA5 issue and not for instance same issue for MERRA-2 data:

• Error message:

  /raid-manaslu/maref/Software/anaconda3/envs/RAIDERC/lib/python3.10/site-packages/RAiDER/utilFcns.py:236: RasterioDeprecationWarning: statistics() will be removed in 2.0.0. Please switch to stats().
  stats = src.statistics(band)
  WARNING: Weather model only extends to the surface topography; height levels below the topography will be interpolated from the surface and may be inaccurate.
  ...
  ERROR: "No variable named 't'. Variables on the dataset include ['z', 'lnsp', 'valid_time', 'model_level', 'latitude', 'expver', 'longitude']"
  ERROR: Weather model files are: []
  ERROR: Downloading and/or preparation of ERA-5 failed.
  
  
  

I have downloaded weather data using cdsapi and I don't underestand why i have such error in raider

>>> import cdsapi
>>> c = cdsapi.Client()
2024-11-11 01:03:47,488 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your 
[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.
2024-11-11 01:03:47,489 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.
2024-11-11 01:03:47,489 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.
2024-11-11 01:03:47,489 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**
2024-11-11 01:03:47,489 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
>>> c.retrieve(
...     'reanalysis-era5-single-levels',
...     {
...         'product_type': 'reanalysis',
...         'variable': ['temperature', 'geopotential', 'lnsp'],
...         'year': '2014',
...         'month': '10',
...         'day': '18',
...         'time': '09:00',
...         'format': 'netcdf'
...     },
...     'weather_files/ERA-5_2014_10_18_T09_44_45.nc')
2024-11-11 01:03:47,892 WARNING [2024-10-10T00:00:00] The final validated ERA5 differs from ERA5T from July 2024 until further notice - please refer to our
[Forum announcement](https://forum.ecmwf.int/t/final-validated-era5-product-to-differ-from-era5t-in-july-2024/6685)
for details and watch it for further updates on this.
2024-11-11 01:03:47,892 INFO Request ID is 7523488e-3256-48f1-8d14-9dfe52a85b3a
2024-11-11 01:03:47,960 INFO status has been updated to accepted
2024-11-11 01:03:50,628 INFO status has been updated to running
2024-11-11 01:03:52,956 INFO status has been updated to successful
'weather_files/ERA-5_2014_10_18_T09_44_45.nc'  

>>> import xarray as xr
>>> ds=xr.load_dataset('weather_files/ERA-5_2014_10_18_T09_44_45.nc', engine='netcdf4')
>>> print(ds)
<xarray.Dataset> Size: 4MB
Dimensions:     (valid_time: 1, latitude: 721, longitude: 1440)
Coordinates:
    number      int64 8B 0
  * valid_time  (valid_time) datetime64[ns] 8B 2014-10-18T09:00:00
  * latitude    (latitude) float64 6kB 90.0 89.75 89.5 ... -89.5 -89.75 -90.0
  * longitude   (longitude) float64 12kB 0.0 0.25 0.5 0.75 ... 359.2 359.5 359.8
    expver      <U4 16B '0001'
Data variables:
    z           (valid_time, latitude, longitude) float32 4MB 0.3228 ... 2.73...
Attributes:
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2024-11-10T03:14 GRIB to CDM+CF via cfgrib-0.9.1...
>>> 

I really appreciate if you can help me for this to fix that?

Best regards,
Mohammad

[mohseniaref]

Thanks for update @jlmaurer .What would be possible solution?

[jlmaurer]

Hi @mohseniaref CDS updated their API and I think that's what is causing the issue. We'll have to fix the API calls from RAiDER to match the new requirements. I'm looking at here and here. I'll start working on this but just taking a quick look it may take a bit of time to figure out what has been updated.

Letting @dbekaert and @bbuzz31 know what's up.