trouble creating EAM initial conditions

[nanr]

I am having trouble creating EAM initial conditions from ERA5 files that I've downloaded from CDS using HICCUP tools.

Notes: Source Code adapted from: template_hiccup_scripts/create_EAM_IC_from_ERA5-NOAA.pycreate_EAM_IC_from_ERA5-NOAA.py

Question: Is it possible that I've erroneously downloaded netcdf-4 instead of 64bit. I have successfully created EAM ICs before. However, this is the first time (at least that I have noticed) that the input files download from CDS as netcdf-4 files.

Error:

Horizontally remapping the multi-file data to temporary files...

ncremap --nco_opt='-O --no_tmp_fl --hdr_pad=100000' --map_file=/pscratch/sd/n/nanr/ERA5_init/map_721x1440_to_ne30np4.nc --in_file=/pscratch/sd/n/nanr/ERA5_init/ERA5.atm.1970-01-05.00.nc --out_file=/pscratch/sd/n/nanr/ERA5_init/tmp_atm_data.ne30np4.L72.T.20250104.200449.nc --var_lst=t,latitude,longitude --fl_fmt=64bit_data
Input #00: /pscratch/sd/n/nanr/ERA5_init/ERA5.atm.1970-01-05.00.nc
Map/Wgt : /pscratch/sd/n/nanr/ERA5_init/map_721x1440_to_ne30np4.nc
ERROR: nco_def_var() failed to nc_def_var() variable "expver"
ERROR NC_BADTYPE Not a netCDF data type
HINT: NC_EBADTYPE errors can occur when NCO tries to write netCDF4 features to a netCDF3 file, and, separately, when a variable's type conflicts with its _FillValue type. 1. netCDF4 features that cannot be defined in a netCDF3 file, and that thus will trigger this error, include groups and netCDF4 atomic types (e.g., NC_STRING, NC_UBYTE, NC_INT64). To avoid this, remove all netCDF4 features before attempting to convert them to netCDF3 format, or write a netCDF4 output file instead of a netCDF3 output file. 2. The netCDF best practice is for _FillValue attributes to be the same numeric type as their host variables.
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_def_var()
nco_err_exit(): ERROR Error code is -45. Translation into English with nc_strerror(-45) is "NetCDF: Not a valid data type or _FillValue type mismatch"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
ncks -O -t 2 --fl_fmt=64bit_data -O --no_tmp_fl --hdr_pad=100000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=login14 --gaa remap_version=5.3.0 -v t,latitude,longitude --rgr lat_nm_out=lat --rgr lon_nm_out=lon --map_fl="/pscratch/sd/n/nanr/ERA5_init/map_721x1440_to_ne30np4.nc" "/pscratch/sd/n/nanr/ERA5_init/ERA5.atm.1970-01-05.00.nc" "/pscratch/sd/n/nanr/ERA5_init/tmp_atm_data.ne30np4.L72.T.20250104.200449.nc"
Traceback (most recent call last):
File "/global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/HICCUP/create_EAM_IC_from_ERA5-NOAA.S2S.py", line 107, in
hiccup_data.remap_horizontal_multifile(file_dict)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/HICCUP/hiccup/hiccup_data_class.py", line 701, in remap_horizontal_multifile
run_cmd(cmd,verbose,shell=True)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/HICCUP/hiccup/hiccup_utilities.py", line 24, in run_cmd
sp.check_call(cmd,shell=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/global/homes/n/nanr/.conda/envs/hiccup_env_2025/lib/python3.13/subprocess.py", line 419, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ncremap --nco_opt='-O --no_tmp_fl --hdr_pad=100000' --map_file=/pscratch/sd/n/nanr/ERA5_init/map_721x1440_to_ne30np4.nc --in_file=/pscratch/sd/n/nanr/ERA5_init/ERA5.atm.1970-01-05.00.nc --out_file=/pscratch/sd/n/nanr/ERA5_init/tmp_atm_data.ne30np4.L72.T.20250104.200449.nc --var_lst=t,latitude,longitude --fl_fmt=64bit_data' returned non-zero exit status 1.

[whannah1]

@nanr thanks for opening an issue. I also encountered this recently and I've been working with Charlie Zender on a fix. The problem is a bit complicated to explain, but basically it's trying to copy and netcdf4 string type variable to a netcdf3 file. For various reasons I'd like to keep the current HICCUP workflow that uses the netcdf3 format for most operations unchanged. Charlie has a fix that converts the string into a char array so that it can persist across the remapping step when the netcdf3 file type is requested. I think this feature is still in a dev branch and not in the release version of NCO.

I tried to modify the NCO commands in HICCUP to explicitly skip this variable during the horizontal remap step. However, this doesn't work because the variable exclusion flags only apply to variables actively being remapped, so NCO still attempts to copy "expver" to the destination file.

Luckily, there's a simple workaround to simply remove the "expver" variable from the ERA5 data files while we wait for the next release of NCO:

ncks -O -C -x -v expver ${FILE1} ${FILE2}

If $FILE1==$FILE2 the file will be overwritten, but it's generally "safer" to make it write to a new file. Either way, this will address the error until the next release of NCO.

[nanr]

Thanks for the quick response @whannah1 ! I'll give it a try!