Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add capability to run diff resolutions for marine anl and background #3238

Open
wants to merge 31 commits into
base: develop
Choose a base branch
from

Conversation

guillaumevernieres
Copy link
Contributor

@guillaumevernieres guillaumevernieres commented Jan 16, 2025

Description

This PR mostly includes configuration changes to allow the use a B-matrix at a defined resolution that may or may not be the same as the background.

Changes of note:

Dependencies

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

Subset of the ci tests on hercules and hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

Copy link
Contributor

@DavidNew-NOAA DavidNew-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good except for the one comment

sorc/link_workflow.sh Outdated Show resolved Hide resolved
@guillaumevernieres guillaumevernieres added the JEDI Feature development to support JEDI-based DA label Jan 23, 2025
DavidNew-NOAA
DavidNew-NOAA previously approved these changes Jan 23, 2025
@RussTreadon-NOAA RussTreadon-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Feb 8, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Feb 8, 2025
@RussTreadon-NOAA RussTreadon-NOAA removed this from the GFS v17 milestone Feb 8, 2025
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Feb 8, 2025
@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C48_S2SW FAILED on Hercules in Build# 1 with error logs:

/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C48_S2SW_346861aa/logs/2021032312/gfs_wavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Feb 8, 2025
@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C48_S2SW FAILED on Hercules in Build# 1 in
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/EXPDIR/C48_S2SW_346861aa

@RussTreadon-NOAA
Copy link
Contributor

RussTreadon-NOAA commented Feb 8, 2025

Hercules C48_S2SW failure

The Hercules failure may be a system issue. The log file ends with a tmpcmdfile.010: Stale file handle error message.

+ exgfs_wave_post_pnt.sh[405]: sed 's/^\(.*\)$/\/work2\/noaa\/global\/CI\/HERCULES\/3238\/global-workflow\/ush\/wave_outp_spec.sh \1 2021032322 bull \/work\/noaa\/stmp\/mterry\/HERCULES\/RUNDIRS\/C48_S2SW_346861aa\/gfs.2021032312\/wavepostpnt.1186317\/output_20210323220000 > \/work\/noaa\/stmp\/mterry\/HERCULES\/RUNDIRS\/C48_S2SW_346861aa\/gfs.2021032312\/wavepostpnt.1186317\/output_20210323220000\/bull_\1.out 2>\&1/' buoy_lst.txt
/work2/noaa/global/CI/HERCULES/3238/global-workflow/scripts/exgfs_wave_post_pnt.sh: line 405: tmpcmdfile.010: Stale file handle
+ exgfs_wave_post_pnt.sh[1]: postamble exgfs_wave_post_pnt.sh 1739043891 1

@RussTreadon-NOAA RussTreadon-NOAA added the CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion label Feb 8, 2025
@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 1 with error logs:

/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_346861aa/logs/2021032500/gfs_wavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 1 in
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_346861aa

@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 1 with error logs:

/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C96C48_hybatmDA_346861aa/logs/2021122106/enkfgdas_fcst_mem001.log
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C96C48_hybatmDA_346861aa/logs/2021122106/enkfgdas_fcst_mem002.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Feb 8, 2025

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 1 in
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/EXPDIR/C96C48_hybatmDA_346861aa

@RussTreadon-NOAA
Copy link
Contributor

RussTreadon-NOAA commented Feb 8, 2025

Hercules C48mx500_3DVarAOWCDA failure

According to /work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_346861aa/logs/2021032500/gfs_wavepostpnt.log, job gfs_wavepostpnt failed because the run directory did not exist.

Begin exgfs_wave_post_pnt.sh at Sat Feb  8 20:28:15 UTC 2025
+ exgfs_wave_post_pnt.sh[43]: cd /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48mx500_3DVarAOWCDA_346861aa/gfs.2021032500/wavepostpnt.1366095
/work2/noaa/global/CI/HERCULES/3238/global-workflow/scripts/exgfs_wave_post_pnt.sh: line 43: cd: /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48mx500_3DVarAOWCDA_346861aa/gfs.2021032500/wavepostpnt.1366095: No such file or directory
+ exgfs_wave_post_pnt.sh[1]: postamble exgfs_wave_post_pnt.sh 1739046495 1

This is indeed true. gfs_cleanup completed before gfs_wavepostpnt began

Begin cleanup.sh at Sat Feb  8 20:27:14 UTC 2025
+ cleanup.sh[7]: . /work2/noaa/global/CI/HERCULES/3238/global-workflow/ush/load_fv3gfs_modules.sh

...

+ exglobal_cleanup.sh[118]: rm -rf /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48mx500_3DVarAOWCDA_346861aa/gfs.2021032500
+ exglobal_cleanup.sh[120]: echo 'Cleanup /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48mx500_3DVarAOWCDA_346861aa/gfs.2021032500 completed!'
Cleanup /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48mx500_3DVarAOWCDA_346861aa/gfs.2021032500 completed!
+ exglobal_cleanup.sh[1]: postamble exglobal_cleanup.sh 1739046449 0
+ preamble.sh[70]: set +x
End exglobal_cleanup.sh at 20:27:36 with error code 0 (time elapsed: 00:00:07)
+ JGLOBAL_CLEANUP[7]: status=0
+ JGLOBAL_CLEANUP[8]: ((  status != 0  ))
+ JGLOBAL_CLEANUP[16]: exit 0
+ JGLOBAL_CLEANUP[1]: postamble JGLOBAL_CLEANUP 1739046437 0
+ preamble.sh[70]: set +x
End JGLOBAL_CLEANUP at 20:27:36 with error code 0 (time elapsed: 00:00:19)
+ cleanup.sh[17]: status=0
+ cleanup.sh[19]: exit 0
+ cleanup.sh[1]: postamble cleanup.sh 1739046434 0
+ preamble.sh[70]: set +x
End cleanup.sh at 20:27:36 with error code 0 (time elapsed: 00:00:22)

gfs_wavepostpnt began at

Begin wavepostpnt.sh at Sat Feb  8 20:27:15 UTC 2025

gfs_cleanup ended at

End cleanup.sh at 20:27:36 with error code 0 (time elapsed: 00:00:22)

Seems the job dependencies in C48mx500_3DVarAOWCDA_346861aa.xml are not correct.

@RussTreadon-NOAA
Copy link
Contributor

Hercules C96C48_hybatmDA failure
Two jobs died

202112210600    enkfgdas_fcst_mem001                     4480918                DEAD                 137         2          95.0
202112210600    enkfgdas_fcst_mem002                     4480919                DEAD                 137         2          95.0

Checks of the enkfgdas forecast log files find

1:
 1: FATAL from PE     1: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability
 1:
 4:
 4: FATAL from PE     4: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability
 4:

in both the mem001 and mem002 log files.

A check of enkfgdas.t06z.enkfstat for enkfgdas.20211221 finds non-phyiscal ensemble mean increments

0:  time level            1
 0:  --------------
 0: ens. mean anal. increment min/max  u   -7955.15087891        9109.36718750
 0: ens. mean anal. increment min/max  v   -11005.5273438        21620.9921875
 0: ens. mean anal. increment min/max  tv   -3555.85644531        5530.83740234
 0: ens. mean anal. increment min/max  q   -1.18051946163        1.93281650543
 0: ens. mean anal. increment min/max  oz  -0.115419737995E-02   0.837389379740E-03
 0: ens. mean anal. increment min/max  ps   -478.353485107        568.966064453
 0:  time level            2
 0:  --------------
 0: ens. mean anal. increment min/max  u   -7837.00927734        8904.18261719
 0: ens. mean anal. increment min/max  v   -10189.2539062        18944.6933594
 0: ens. mean anal. increment min/max  tv   -3640.04980469        7101.26171875
 0: ens. mean anal. increment min/max  q   -1.32147753239        1.35699558258
 0: ens. mean anal. increment min/max  oz  -0.914561096579E-03   0.874249788467E-03
 0: ens. mean anal. increment min/max  ps   -1018.10217285        464.135650635
 0:  time level            3
 0:  --------------
 0: ens. mean anal. increment min/max  u   -9600.04296875        14130.4267578
 0: ens. mean anal. increment min/max  v   -9677.24414062        15389.0644531
 0: ens. mean anal. increment min/max  tv   -3425.51220703        6619.34521484
 0: ens. mean anal. increment min/max  q   -2.04675769806       0.983482599258
 0: ens. mean anal. increment min/max  oz  -0.791968661360E-03   0.897021149285E-03
 0: ens. mean anal. increment min/max  ps   -1133.32531738        668.867553711

C96C48_hybatmDA is a GSI-based atmosphere only CI case. This PR does not alter files used by GSI DA.

Is Hercules experience compute node or file system issues?

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 8, 2025
@emcbot
Copy link

emcbot commented Feb 8, 2025

CI Failed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3238


Experiment C48_ATM_346861aa Completed 1 Cycles: *SUCCESS* at Sat Feb  8 13:45:33 CST 2025
Experiment C48_S2SW_346861aa Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Sat Feb  8 13:51:44 CST 2025
Experiment C48_S2SW_346861aa Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C48_S2SW_346861aa/logs/2021032312/gfs_wavepostpnt.log
Experiment C48mx500_hybAOWCDA_346861aa Completed 2 Cycles: *SUCCESS* at Sat Feb  8 14:09:53 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_346861aa Completed 1 Cycles: *SUCCESS* at Sat Feb  8 14:09:54 CST 2025
Experiment C48mx500_3DVarAOWCDA_346861aa Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Sat Feb  8 14:34:16 CST 2025
Experiment C48mx500_3DVarAOWCDA_346861aa Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_346861aa/logs/2021032500/gfs_wavepostpnt.log
Experiment C96C48_hybatmDA_346861aa Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Sat Feb  8 14:40:12 CST 2025
Experiment C96C48_hybatmDA_346861aa Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C96C48_hybatmDA_346861aa/logs/2021122106/enkfgdas_fcst_mem001.log
/work2/noaa/global/CI/HERCULES/3238/RUNTESTS/COMROOT/C96C48_hybatmDA_346861aa/logs/2021122106/enkfgdas_fcst_mem002.log
Experiment C96_atm3DVar_346861aa Completed 3 Cycles: *SUCCESS* at Sat Feb  8 15:10:23 CST 2025
Experiment C48_S2SWA_gefs_346861aa Completed 1 Cycles: *SUCCESS* at Sat Feb  8 16:29:26 CST 2025

@RussTreadon-NOAA
Copy link
Contributor

Hera g-w CI

Automated Hera g-w CI appears to be broken. Although the CI-Hera-Ready label was applied, the label was never updated to -Building or -Running.

Manually install guillaumevernieres:feature/lowres-b at 346861a

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_ATM_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Feb 08 2025 19:30:26    Feb 08 2025 20:50:19

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_S2SW_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Feb 08 2025 19:30:28    Feb 08 2025 22:35:21

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_S2SWA_gefs_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Feb 08 2025 19:30:29    Feb 08 2025 22:40:20

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Feb 08 2025 19:30:31    Feb 08 2025 19:43:30
202103250000        Done    Feb 08 2025 19:30:31    Feb 08 2025 23:05:21

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_hybAOWCDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Feb 08 2025 19:30:32    Feb 08 2025 19:43:33
202103250000        Done    Feb 08 2025 19:30:32    Feb 08 2025 20:50:26

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Feb 08 2025 19:30:34    Feb 08 2025 19:43:37
202112210000        Done    Feb 08 2025 19:30:34    Feb 08 2025 21:45:32
202112210600        Done    Feb 08 2025 19:30:34    Feb 08 2025 21:30:33

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Feb 08 2025 19:30:35    Feb 08 2025 19:55:28
202112201800        Done    Feb 08 2025 19:30:35    Feb 08 2025 22:10:27
202112210000        Done    Feb 08 2025 19:30:35    Feb 08 2025 21:58:09

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Feb 08 2025 19:30:36    Feb 08 2025 19:43:44
202402240000        Done    Feb 08 2025 19:30:36    Feb 08 2025 22:35:28
202402240600        Done    Feb 08 2025 19:30:36    Feb 08 2025 22:20:29

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96_S2SWA_gefs_replay_ics_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202011010000        Done    Feb 08 2025 19:30:38    Feb 08 2025 20:45:36

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96_atm3DVar_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Feb 08 2025 19:30:39    Feb 08 2025 19:43:53
202112210000        Done    Feb 08 2025 19:30:39    Feb 08 2025 22:10:31
202112210600        Done    Feb 08 2025 19:30:39    Feb 08 2025 21:45:38

All jobs successfully ran to completion on Hera.

@RussTreadon-NOAA
Copy link
Contributor

Since Hera g-w CI was manually run and passed, remove CI-Hera-Ready label

@RussTreadon-NOAA RussTreadon-NOAA removed the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Feb 9, 2025
@RussTreadon-NOAA
Copy link
Contributor

Orion g-w CI

Automated Orion g-w CI appears to be broken. Although the CI-Orion-Ready label was applied, the label was never updated to -Building or -Running.

Manually install guillaumevernieres:feature/lowres-b at 346861a

The following CI cases have dead jobs

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48mx500_3DVarAOWCDA_pr3238
202103250000         gfs_wavepostpnt                    19631378                DEAD                   1         2          39.0
202103250000             gfs_cleanup                    19631379                DEAD                   1         2          41.0

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_S2SW_pr3238
202103231200         gfs_wavepostpnt                    19630936                DEAD                   1         2          40.0
202103231200             gfs_cleanup                    19630937                DEAD                   1         2          40.0

These failures are similar to what was observed on Hercules.

  • C48mx500_3DVarAOWCDA: gfs_cleanup job began 3 seconds after gfs_wavepostpnt
    • gfs_wavepostpnt aborted with cannot open 'tmpcmdfile.013' for reading: No such file or directory
    • gfs_cleanup aborted with rm: cannot remove '/work/noaa/stmp/rtreadon/ORION/RUNDIRS/C48mx500_3DVarAOWCDA_pr3238/gfs.2021032500/wavepostpnt.3308877': Directory not empty
  • C48_S2SW: gfs_cleanup job began 3 seconds after gfs_wavepostpnt
    • gfs_wavepostpnt aborted with cannot open 'tmpcmdfile.013' for reading: No such file or directory
    • gfs_cleanup aborted with rm: cannot remove '/work/noaa/stmp/rtreadon/ORION/RUNDIRS/C48_S2SW_pr3238/gfs.2021032312/wavepostpnt.2497596': Directory not empty

Hercules and Orion CI differ from Hera in that HPSSARCH="NO" on these machines. g-w CI writes HPSS tarballs on Hera. This does not happen on Hercules or Orion. The Hera gfs_cleanup has a dependency on gfs_arch_tars. The Hercules and Orion gfs_cleanup does not have this dependency. Thus, on Hercules and Orion gfs_cleanup starts sooner. Unfortunately, gfs_wavepostpnt is running alongside gfs_cleanup. For those CI cases in which gfs_wavepostpnt is run we need to add this job dependency to gfs_cleanup.

Other g-w CI cases successfully ran to completion on Orion

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_ATM_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Feb 08 2025 22:58:50    Feb 09 2025 00:10:04

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48mx500_hybAOWCDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Feb 08 2025 22:58:54    Feb 08 2025 23:15:09
202103250000        Done    Feb 08 2025 22:58:54    Feb 09 2025 00:35:08

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_S2SWA_gefs_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Feb 08 2025 22:58:56    Feb 09 2025 02:04:32

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96_atm3DVar_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Feb 08 2025 22:59:00    Feb 08 2025 23:15:13
202112210000        Done    Feb 08 2025 22:59:00    Feb 09 2025 02:00:13
202112210600        Done    Feb 08 2025 22:59:00    Feb 09 2025 01:35:11

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_hybatmaerosnowDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Feb 08 2025 22:59:02    Feb 08 2025 23:20:20
202112201800        Done    Feb 08 2025 22:59:02    Feb 09 2025 01:38:58
202112210000        Done    Feb 08 2025 22:59:02    Feb 09 2025 01:37:10

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Feb 08 2025 22:59:04    Feb 08 2025 23:15:18
202112210000        Done    Feb 08 2025 22:59:04    Feb 09 2025 01:09:48
202112210600        Done    Feb 08 2025 22:59:04    Feb 09 2025 01:32:48

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_ufs_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Feb 08 2025 22:59:05    Feb 08 2025 23:15:20
202402240000        Done    Feb 08 2025 22:59:05    Feb 09 2025 02:45:18
202402240600        Done    Feb 08 2025 22:59:05    Feb 09 2025 03:00:18

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96_S2SWA_gefs_replay_ics_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202011010000        Done    Feb 08 2025 22:59:07    Feb 09 2025 00:25:23

@RussTreadon-NOAA
Copy link
Contributor

Since Orion g-w CI was manually run, remove CI-Orion-Ready label

@RussTreadon-NOAA RussTreadon-NOAA removed the CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion label Feb 9, 2025
@RussTreadon-NOAA
Copy link
Contributor

g-w CI summary

g-w CI successfully ran all jobs Hera and WCOSS2 (Dogwood). g-w CI jobs fail on Hercules and Orion. There are similarities between the Hercules and Orion failures.

The Hercules enkf.x failure in C96C48_hybatmDA is odd. Non-physical results are generated. Since enkf.x works OK on other platforms, the Hercules failure appears to be machine specific. Compiler and environment variable settings should be examined. It's also possible that the Hercules enkf.x failure represents a filesystem issue.

Label based automated g-w CI on Hera and Orion is not working. The Ready label did not trigger CI on these machines. The Ready label successfully triggered Hera g-w CI.

How shall we proceed? The g-w CI failures noted above do not appear to be related to changes in this PR.

Tagging @WalterKolczynski-NOAA , @aerorahul , @TerrenceMcGuinness-NOAA , @guillaumevernieres , @DavidNew-NOAA for awareness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed JEDI Feature development to support JEDI-based DA
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement the marine analysis at a resolution different from the deterministic Background error at 1/2 deg
9 participants