multiple netcdf_parallel tests fail on hercules #2015

DeniseWorthen · 2023-11-29T13:52:59Z

Description

The control_wrtGauss_netcdf_parallel_intel fails on hercules (intel) with the following error:

2023-11-29 11:42:48.544357 +0000 ERROR /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0-rc1/cache/build_stage/spack-s\
tage-nccmp-1.9.0.1-4n5sfwacmwzksu4hkop5vwvjpqowwa3o/spack-src/src/nccmp_data.c:3675 NetCDF: HDF error

which results in

 Comparing atmf024.nc ............ALT CHECK......ERROR

To Reproduce:

Attempt to run this test on Hercules, check the atmf024.nc_nccmp.log file in the run directory for the error message.

Additional context

The issue was first noted in PR #1990.

Output

The text was updated successfully, but these errors were encountered:

BrianCurtis-NOAA · 2023-11-29T14:51:31Z

@DeniseWorthen Do the files actually differ, or is NCCMP getting hung up with one of the options we use in the call to NCCMP?

DeniseWorthen · 2023-11-29T14:58:26Z

Unknown. It is failing comparison of that single file with "HDF error", as posted.

junwang-noaa · 2023-11-29T15:19:50Z

Can we compare the two files manually using nccmp? I am wondering if it is file difference or it is nccmp that causes the issue.

DeniseWorthen · 2023-11-29T15:20:42Z

Transferring the baseline atmf024.nc file and the output from a failed RT case to hera and comparing them with

nccmp -d -S -q -f -B --Attribute=checksum --warn=format

also produces an error:

2023-11-29 15:13:22.378912 +0000 ERROR nccmp_data.c:3677 NetCDF: HDF error

On Hera, trying to simply dump each of these files to a cdl file produces an error for the baseline file, but not the RT test file:

ncdump atmf024.nc >atmf024.cdl
NetCDF: HDF error
Location: file vardata.c; fcn print_rows line 478

I suspect it is the baseline file which is bad.

BrianCurtis-NOAA · 2023-11-29T15:30:34Z

@jkbk2004 Are you using cp or rsync for the RDHPCS machines to copy baselines to the baseline storage area?

jkbk2004 · 2023-11-29T16:10:32Z

@jkbk2004 Are you using cp or rsync for the RDHPCS machines to copy baselines to the baseline storage area?

mix use. in this case, files are identical with experiment but nccmp issue

junwang-noaa · 2023-11-29T16:17:33Z

@jkbk2004 Please see the results from Denise.

On Hera, trying to simply dump each of these files to a cdl file produces an error for the baseline file, but not the RT test file:

ncdump atmf024.nc >atmf024.cdl
NetCDF: HDF error
Location: file vardata.c; fcn print_rows line 478

It looks to me. The issue is in the baseline file, not nccmp.

jkbk2004 · 2023-11-29T16:20:29Z

I will setup cases to confirm on both hera and hercules.

DeniseWorthen · 2023-11-29T16:22:14Z

Just to note--I copied the files from hercules to hera and the report above is for using nccmp or ncdump on Hera. This was in case there was a problem w/ Hercules' nccmp version.

junwang-noaa · 2023-11-29T16:31:34Z

Thanks, Denise. That is what we want to confirm, if the comparison fails because of the nccmp or because of the file issue. @jkbk2004 I think you only need to check the baseline on Hecules.

DusanJovic-NOAA · 2023-11-29T17:24:54Z

Which files are we talking about? These two:

/work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_3114426/control_wrtGauss_netcdf_parallel_intel/atmf000.nc

and

/work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231122/control_wrtGauss_netcdf_parallel_intel/atmf000.nc

jkbk2004 · 2023-11-29T17:36:39Z

@DusanJovic-NOAA I am checking with /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_175880/control_wrtGauss_netcdf_parallel_intel/atmf024.nc

DeniseWorthen · 2023-11-29T17:44:55Z

@DusanJovic-NOAA The file that fails w/ the hdf error is for the atmf024.nc file.

jkbk2004 · 2023-11-29T19:34:45Z

@zach1221 nccmp/hercules compares ok /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_166684/control_wrtGauss_netcdf_parallel_intel/atmf024.nc atmf024.nc and /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231122/control_wrtGauss_netcdf_parallel_intel/atmf024.nc
but fails with /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_175880/control_wrtGauss_netcdf_parallel_intel/atmf024.nc
@DeniseWorthen @junwang-noaa ncdump conversion to cdl works ok with both baseline one and rt_166684 but not rt_175880. It looks hercules system issue

zach1221 · 2023-12-01T21:08:17Z

@DeniseWorthen I finally got around to testing your idea, regarding disk space. I cleaned out my experiment directories in stmp, and re-ran the control_wrtGauss cases on again with ecflow. With an emptied space they pass consistently.

DeniseWorthen · 2023-12-01T21:28:20Z

@zach1221 Really interesting, thanks. I wonder why this works!? I did check the file sizes for the atmf files and they weren't that large. Hm.

DeniseWorthen · 2023-12-06T22:04:42Z

I've updated the title of this Issue. I just ran on Hercules with the CICE PR branch and got multiple failures with "ALT CHECK ERROR" for various tests:

Checking test 034 control_wrtGauss_netcdf_parallel_intel results ....
Comparing atmf024.nc ............ALT CHECK......ERROR

Checking test 060 regional_netcdf_parallel_intel results ....
Comparing phyf000.nc ............ALT CHECK......ERROR

Checking test 085 control_wrtGauss_netcdf_parallel_debug_intel results
Comparing atmf000.nc ............ALT CHECK......ERROR

Checking test 126 conus13km_debug_qr_intel results ....
Comparing RESTART/20210512.170000.fv_core.res.tile1.nc ............ALT CHECK......ERROR

DeniseWorthen · 2023-12-07T02:13:16Z

Running these four tests again gave a pass for the conus13 and the netcdf_parallel_debug. The other two again had ERRORS, but on different .nc files. These tests appear to be unstable on hercules.

DusanJovic-NOAA · 2023-12-07T12:51:41Z

Running these four tests again gave a pass for the conus13 and the netcdf_parallel_debug. The other two again had ERRORS, but on different .nc files. These tests appear to be unstable on hercules.

I those cases that fail, do you (always) see 'HDF error'?

DeniseWorthen · 2023-12-07T13:10:54Z

@DusanJovic-NOAA Yes, the failed cases seem to always show

ERROR /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0-rc1/cache/build_stage/spack-st
age-nccmp-1.9.0.1-4n5sfwacmwzksu4hkop5vwvjpqowwa3o/spack-src/src/nccmp_data.c:3675 NetCDF: HDF error

My run directory is

/work2/noaa/stmp/dworthen/stmp/dworthen/FV3_RT/rt_3821674/

DusanJovic-NOAA · 2023-12-07T14:14:35Z

It seems the file /work2/noaa/stmp/dworthen/stmp/dworthen/FV3_RT/rt_3821674/control_wrtGauss_netcdf_parallel_intel/atmf000.nc is either corrupted, or there's a bug in HDF5 library, or both.

When I run ncdump I see:

$ ncdump /work2/noaa/stmp/dworthen/stmp/dworthen/FV3_RT/rt_3821674/control_wrtGauss_netcdf_parallel_intel/atmf000.nc
ncdump: /work2/noaa/stmp/dworthen/stmp/dworthen/FV3_RT/rt_3821674/control_wrtGauss_netcdf_parallel_intel/atmf000.nc: NetCDF: HDF error

Same error message we see when we run nccmp. However when I run h5dump, I see:

$ h5dump /work2/noaa/stmp/dworthen/stmp/dworthen/FV3_RT/rt_3821674/control_wrtGauss_netcdf_parallel_intel/atmf000.nc
h5dump error: internal error (file /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0-rc1/cache/build_stage/spack-stage-hdf5-1.14.0-4qmsxztujdfpvzjzay4dyr2d2vxd352n/spack-src/tools/src/h5dump/h5dump.c:line 1525)
HDF5: infinite loop closing library
      L,T_top,P,P,Z,FD,VL,VL,PL,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL

I'm not sure how to interpret this error message, HDF5 library bug, MPI/compiler bug, system/filesystem issue or something else.

junwang-noaa · 2023-12-07T14:18:36Z

@DeniseWorthen @DusanJovic-NOAA So far I noticed this error only happened on Hercules. We have hdf5 1.14.0 installed on other platforms for a while and I have never heard the error. I think we need to report the problem to Hercules system admin.

jkbk2004 · 2023-12-07T14:26:01Z

@climbfuji @ulmononian FYI: some issues with HDF/nccmp on hercules. Random failures when it writes nc files.

climbfuji · 2023-12-07T18:30:50Z

@ulmononian Can EPIC look into this?

DeniseWorthen · 2023-12-08T16:23:45Z

@DusanJovic-NOAA I went back and looked at the all the hercules logs. The first time that the control_wrtGauss_netcdf_parallel_intel has the alt-check error is in your PR #1990

ufs-weather-model/tests/logs/RegressionTests_hercules.log

Line 1976 in f6918a1

    
           baseline dir = /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231122/control_wrtGauss_netcdf_parallel_intel

DusanJovic-NOAA · 2023-12-08T16:31:08Z

@DusanJovic-NOAA I went back and looked at the all the hercules logs. The first time that the control_wrtGauss_netcdf_parallel_intel has the alt-check error is in your PR #1990

ufs-weather-model/tests/logs/RegressionTests_hercules.log

Line 1976 in f6918a1

baseline dir = /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231122/control_wrtGauss_netcdf_parallel_intel

That's the PR in which we added netcdf quantization. Maybe that is somehow triggering this HDF error, but why only on Hercules?

DusanJovic-NOAA · 2023-12-13T17:11:23Z

@lisa-bengtsson I believe the parallel compression issue is known with hdf5 1.10.6 in the workflow PR you referred, it happened on several platforms. We have moved hdf 1.14.0, which resolved the issue, see https://www.hdfgroup.org/2022/12/release-of-hdf5-1-14-0-newsletter-189/.

The problem here only happened on Hercules, the same tests passed on all other platforms.

@DusanJovic-NOAA is the regional test you ran using lossy compression or lossless compression?

Lossless (deflate).

DusanJovic-NOAA · 2023-12-13T18:58:10Z

I'm not sure if it's a long shot, but would it be convenient to try running where the output files being corrupted can be written directly to a non-lustre space, like /tmp or $HOME or something? EDIT: Just googling "lustre"+"hdf error" I'm seeing a few instances of users running into similar issues, so if it's possible to test writing to a non-lustre space it could help to rule out filesystem issues.

I changed the write routine to create output files in my home directory:

-          ncerr = nf90_create(trim(filename),&
+          ncerr = nf90_create("/home/djovic/hdf_out/"//trim(filename),&

and ran regional_netcdf_parallel test couple times (I think 3 or 4 times). After each run, I ran h5dump on each history output:

$ h5dump dynf000.nc > /dev/null
$ h5dump dynf003.nc > /dev/null
... etc

and all dumps were successful, i.e. no 'HDF error' or 'h5dump error: unable to print data'.

climbfuji · 2023-12-13T19:21:35Z

I think someone needs to bring this up to the sysadmins. Please let me know if anyone wants to do that, otherwise I can take care of it.

junwang-noaa · 2023-12-13T19:52:33Z

I created an ticket to Hercules helpdesk.
@DusanJovic-NOAA I want to confirm with you: 1) the model version in your test run under /home is e053209 2).The test you are running is regional_netcdf_parallel. would you please confirm?

DusanJovic-NOAA · 2023-12-14T14:00:22Z

I created an ticket to Hercules helpdesk. @DusanJovic-NOAA I want to confirm with you: 1) the model version in your test run under /home is e053209 2).The test you are running is regional_netcdf_parallel. would you please confirm?

Yes. For that test in which the history files were created i my home directory I used e053209 and I ran regional_netcdf_parallel

DusanJovic-NOAA · 2023-12-20T13:03:05Z

Hercules sysadmin suggests that we add export I_MPI_EXTRA_FILESYSTEM=ON to the job card.

diff --git a/tests/fv3_conf/fv3_slurm.IN_hercules b/tests/fv3_conf/fv3_slurm.IN_hercules
index 30ea2981..c4853fb5 100644
--- a/tests/fv3_conf/fv3_slurm.IN_hercules
+++ b/tests/fv3_conf/fv3_slurm.IN_hercules
@@ -36,8 +36,10 @@ export OMP_NUM_THREADS=@[THRD]
 export ESMF_RUNTIME_PROFILE=ON
 export ESMF_RUNTIME_PROFILE_OUTPUT="SUMMARY"
 
-# For mvapich2
-if [[ @[RT_COMPILER] == gnu ]]; then
+if [[ @[RT_COMPILER] == intel ]]; then
+  export I_MPI_EXTRA_FILESYSTEM=ON
+elif [[ @[RT_COMPILER] == gnu ]]; then
+  # For mvapich2
   export MV2_SHMEM_COLL_NUM_COMM=128
 fi

With this change I created new baselines for 4 tests that were frequently failing (control_CubedSphereGrid_parallel, control_wrtGauss_netcdf_parallel, regional_netcdf_parallel and control_wrtGauss_netcdf_parallel_debug) and then ran the regression test for these tests against newly created baselines. All tests passed. I ran the test 2 times so far and I'm going to run it few more times to make sure it consistently passes without crashing nccmp with HDF error.

jkbk2004 · 2023-12-20T13:27:42Z

@DusanJovic-NOAA thanks for the update. We can follow up with the next pr, #2044

jkbk2004 · 2023-12-20T13:28:23Z

@zach1221 @FernandoAndrade-NOAA FYI

edwardhartnett · 2023-12-20T13:29:27Z

It's important that the HDF5 tests are run, and that the netCDF-4 parallel tests are run. If these tests do not pass, then there is a problem with the machine or the install of netCDF. If the HDF5 and netCDF-4 parallel tests pass, then the problem is likely in user code.

Have these tests been run on hercules? Who did the install there? @AlexanderRichert-NOAA do you know if the netCDF and HDF5 parallel I/O tests were run on hercules?

climbfuji · 2023-12-20T15:35:54Z

Hercules sysadmin suggests that we add export I_MPI_EXTRA_FILESYSTEM=ON to the job card.
diff --git a/tests/fv3_conf/fv3_slurm.IN_hercules b/tests/fv3_conf/fv3_slurm.IN_hercules
index 30ea2981..c4853fb5 100644
--- a/tests/fv3_conf/fv3_slurm.IN_hercules
+++ b/tests/fv3_conf/fv3_slurm.IN_hercules
@@ -36,8 +36,10 @@ export OMP_NUM_THREADS=@[THRD]
 export ESMF_RUNTIME_PROFILE=ON
 export ESMF_RUNTIME_PROFILE_OUTPUT="SUMMARY"
 
-# For mvapich2
-if [[ @[RT_COMPILER] == gnu ]]; then
+if [[ @[RT_COMPILER] == intel ]]; then
+  export I_MPI_EXTRA_FILESYSTEM=ON
+elif [[ @[RT_COMPILER] == gnu ]]; then
+  # For mvapich2
   export MV2_SHMEM_COLL_NUM_COMM=128
 fi
With this change I created new baselines for 4 tests that were frequently failing (control_CubedSphereGrid_parallel, control_wrtGauss_netcdf_parallel, regional_netcdf_parallel and control_wrtGauss_netcdf_parallel_debug) and then ran the regression test for these tests against newly created baselines. All tests passed. I ran the test 2 times so far and I'm going to run it few more times to make sure it consistently passes without crashing nccmp with HDF error.

Excellent, thanks! I will add this to the hercules intel config in spack-stack, too. So this hopefully solves the Intel problems discussed here. There are still gnu problems discussed elsewhere, correct?

jkbk2004 · 2023-12-20T17:20:12Z

FYI: https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html

…nodi + #2047, #2053, and #2056 (#2044) FV3 diagnostic fixes, CCPP fixes for model crashes, new PR template - UFS: - commit message in PR template (#2053) - fix hercules crashes (#2015) - CMEPS & FV3: Bad data from in CCPP CLM Lake physics scheme caused model crashes - Communicate changes to lake ice (Closes #2055, NOAA-EMC/CMEPS#105, NOAA-EMC/fv3atm#741) - unit mismatch (NOAA-EMC/fv3atm#736) - FV3: correct errors in diagnostic calculations - snodi had weasdi data in it (NOAA-EMC/fv3atm#736) - revisions to RUC LSM snowfall melting and accumulation (NOAA-EMC/fv3atm#739)

TingLei-NOAA · 2024-02-02T19:16:46Z

@edwardhartnett will you have any suggestions on further investigating of this issue? The problem could occur for either nf90_get_var or nf90_put_var ( in the context of a parallel IO (the one of netcdf 4 on top of hdf5))

climbfuji · 2024-02-02T19:22:56Z

I thought this was resolved for Intel with the additonal env variable (see lengthy discussion above). For GNU, please try the alternative spack-stack-1.6.0 [email protected][email protected] installation.

TingLei-NOAA · 2024-02-02T19:29:28Z

Sorry, I asked in a wrong issue. Thanks @climbfuji . The issue i asked for is NOAA-EMC/GSI#684.

edwardhartnett · 2024-02-02T20:02:38Z

Did HDF5 pass tests with --enable-parallel? Did netCDF pass tests with --enable-parallel-tests? Ed

…

On Fri, Feb 2, 2024 at 12:29 PM TingLei-NOAA ***@***.***> wrote: Sorry, I asked in a wrong issue. Thanks @climbfuji <https://github.com/climbfuji> . The issue i asked for is NOAA-EMC/GSI#684 <NOAA-EMC/GSI#684>. — Reply to this email directly, view it on GitHub <#2015 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJIOMMANKFARVHEHWZH4AM3YRU5CLAVCNFSM6AAAAAA77QG43KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRUGU2DOMBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Edward Hartnett CIRES/NOAA EMC

DeniseWorthen added the bug Something isn't working label Nov 29, 2023

DeniseWorthen mentioned this issue Nov 29, 2023

add component defaults, rename templates to .in, rename ufs.configure for application, cleanup tests #2010

Merged

42 tasks

DeniseWorthen changed the title ~~control_wrtGauss_netcdf_parallel fails on hercules~~ multiple netcdf_parallel tests fail on hercules Dec 6, 2023

junwang-noaa mentioned this issue Dec 7, 2023

Enable forecast-only experiments on Hercules NOAA-EMC/global-workflow#2128

Merged

7 tasks

zach1221 mentioned this issue Dec 19, 2023

bug fix: disable concurrency in GFS_phys_time_vary_init NetCDF calls AND #2040 and #2043 #2041

Merged

35 tasks

DavidHuber-NOAA mentioned this issue Dec 20, 2023

Port to Hercules NOAA-EMC/GSI-utils#23

Merged

JacobCarley-NOAA mentioned this issue Dec 20, 2023

Port rrfs-workflow to Hercules NOAA-EMC/rrfs-workflow#172

Merged

8 tasks

climbfuji mentioned this issue Dec 20, 2023

Set I_MPI_EXTRA_FILESYSTEM=ON on Hercules/Intel JCSDA/spack-stack#921

Closed

zach1221 mentioned this issue Dec 21, 2023

CLM Lake pressure is not density and GFS Diagnostics: weasdi is not snodi + #2047, #2053, and #2056 #2044

Merged

35 tasks

zach1221 closed this as completed in #2044 Dec 21, 2023

junwang-noaa added this to Model infrastructure development FY24Q1 Dec 22, 2023

junwang-noaa moved this to Done in Model infrastructure development FY24Q1 Dec 22, 2023

junwang-noaa assigned DeniseWorthen, DusanJovic-NOAA and junwang-noaa Jan 9, 2024

RussTreadon-NOAA mentioned this issue Feb 1, 2024

Upgrade the GSI to Spack-Stack version 1.6.0 NOAA-EMC/GSI#684

Merged

6 tasks

BrianCurtis-NOAA mentioned this issue Apr 15, 2024

g-w jobs fail on Hercules with NetCDF: HDF error NOAA-EMC/global-workflow#2489

Closed

uturuncoglu mentioned this issue Apr 22, 2024

baseline check is not working as expected on Hercules #2245

Closed

multiple netcdf_parallel tests fail on hercules #2015

multiple netcdf_parallel tests fail on hercules #2015

Comments

DeniseWorthen commented Nov 29, 2023 • edited Loading

Description

To Reproduce:

Additional context

Output

BrianCurtis-NOAA commented Nov 29, 2023

DeniseWorthen commented Nov 29, 2023

junwang-noaa commented Nov 29, 2023

DeniseWorthen commented Nov 29, 2023 • edited Loading

BrianCurtis-NOAA commented Nov 29, 2023

jkbk2004 commented Nov 29, 2023

junwang-noaa commented Nov 29, 2023

jkbk2004 commented Nov 29, 2023

DeniseWorthen commented Nov 29, 2023 • edited Loading

junwang-noaa commented Nov 29, 2023 • edited Loading

DusanJovic-NOAA commented Nov 29, 2023

jkbk2004 commented Nov 29, 2023

DeniseWorthen commented Nov 29, 2023

jkbk2004 commented Nov 29, 2023

zach1221 commented Dec 1, 2023

DeniseWorthen commented Dec 1, 2023

DeniseWorthen commented Dec 6, 2023

DeniseWorthen commented Dec 7, 2023

DusanJovic-NOAA commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023 • edited Loading

DusanJovic-NOAA commented Dec 7, 2023

junwang-noaa commented Dec 7, 2023

jkbk2004 commented Dec 7, 2023 • edited Loading

climbfuji commented Dec 7, 2023

DeniseWorthen commented Dec 8, 2023

DusanJovic-NOAA commented Dec 8, 2023

DusanJovic-NOAA commented Dec 13, 2023

DusanJovic-NOAA commented Dec 13, 2023

climbfuji commented Dec 13, 2023

junwang-noaa commented Dec 13, 2023 • edited Loading

DusanJovic-NOAA commented Dec 14, 2023

DusanJovic-NOAA commented Dec 20, 2023

jkbk2004 commented Dec 20, 2023

jkbk2004 commented Dec 20, 2023

edwardhartnett commented Dec 20, 2023

climbfuji commented Dec 20, 2023

jkbk2004 commented Dec 20, 2023

TingLei-NOAA commented Feb 2, 2024

climbfuji commented Feb 2, 2024

TingLei-NOAA commented Feb 2, 2024

edwardhartnett commented Feb 2, 2024 via email

DeniseWorthen commented Nov 29, 2023 •

edited

Loading

DeniseWorthen commented Nov 29, 2023 •

edited

Loading

DeniseWorthen commented Nov 29, 2023 •

edited

Loading

junwang-noaa commented Nov 29, 2023 •

edited

Loading

DeniseWorthen commented Dec 7, 2023 •

edited

Loading

jkbk2004 commented Dec 7, 2023 •

edited

Loading

junwang-noaa commented Dec 13, 2023 •

edited

Loading