ILAMB failing in Unified RC

[forsyth2]

Using source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc1_chrysalis.sh I'm getting a core dump for ILAMB. This error doesn't occur using the last Unified RC.

Full cfg:

[default]
case = v2.LR.historical_0201
constraint = ""
environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc1_chrysalis.sh"
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
input = "/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201"
input_subdir = archive/atm/hist
mapping_file = "/home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_ilamb220rc3v4_output/v2.LR.historical_0201"
partition = "debug"
qos = "regular"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_ilamb220rc3v4_www"

[ts]
active = True
walltime = "00:30:00"
years = "1850:1854:2",

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  ts_fmt = "cmip"

  [[ land_monthly ]]
  extra_vars = "landfrac"
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"
  ts_fmt = "cmip"

[ilamb]
active = True
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc1_chrysalis.sh"
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1850:1854:2",

In /lcrc/group/e3sm/ac.forsyth2/zppy_test_ilamb220rc3v4_output/v2.LR.historical_0201/post/scripts:

• ilamb_1852-1853.o246267 completed successfully (https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_test_ilamb220rc3v4_www/v2.LR.historical_0201/ilamb/180x360_aave_1852-1853/). The log does report Errors occurred in the run, but I believe those are just the variables/plots missing data.
• ilamb_1850-1851.o246266 however fails. srun -N 1 ilamb-run --config ilamb.cfg --model_root $model_root --regions global --model_year ${Y1} 2000 is the line that fails.

Log file output begins with:

/lcrc/group/e3sm/ac.forsyth2/zppy_test_ilamb220rc3v4_output/v2.LR.historical_0201/post/scripts/tmp.246266.ZkaA
/lcrc/group/e3sm/ac.forsyth2/zppy_test_ilamb220rc3v4_output/v2.LR.historical_0201/post/scripts
[chr-0493:623715:0:623715] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x40)

[forsyth2]

@chengzhuzhang Do you have any insights on this? I can't quite tell if it's a problem with how zppy is calling ilamb-run or if ilamb-run itself is the problem in the new Unified RC.

[chengzhuzhang]

Hi @forsyth2 you mentioned This error doesn't occur using the last Unified RC. Would you clarify which Unified RC worked?
There hasn't been a change regarding how ilamb is called. I think this might be an problem how ilamb is build with Unified. This might be something we should report to @xylar

[forsyth2]

Oh sorry, I should have just said last Unified. That commented line -- "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh", meaning Unified 1.7.0.

Actually, I didn't realize the new Unified RC would be released today. I just finished testing ILAMB specifically on that. The above in fact works using source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc5_chrysalis.sh!

[chengzhuzhang]

Great! I'm glad rc5 works okay. Thanks you for testing!