Update for new fix structure #688
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kate didn't open a PR before she left, but I found her stuff and finished it off. I don't have access to Jet to test there. I do not know who maintains fix on the other platforms not already tested or if they have been updated. |
|
@WalterKolczynski-NOAA S4 has the updated fix directories. I will run a cycled and S2S test on that system. I can run the UFS_UTILS regression tests on Jet, but the global workflow port has not been updated since March, so I won't be able to run a cycled or coupled test at this time. Would you like me to go ahead with the RTs? |
|
@WalterKolczynski-NOAA I can run the UFS_UTILS regression tests. |
|
@WalterKolczynski-NOAA gdas_init is looking for the mosaic files in the wrong location. The chgres scripts all set |
|
@DavidHuber-NOAA You won't be able to test workflow because I haven't opened a PR with the changes there. It is waiting on this PR before I open it. I don't think another workflow test is necessary for this PR. |
|
@WalterKolczynski-NOAA OK, that makes sense, thanks. |
|
@WalterKolczynski-NOAA I will run RTs on S4 instead. |
WalterKolczynski-NOAA
added a commit
to WalterKolczynski-NOAA/global-workflow
that referenced
this pull request
Aug 31, 2022
Removes all of the code and scripts that were moved to the new gfs-utils repo and adjusts workflow scripts to build and use them from the new location. Some of the build scripts had unnecessary calls to machine-setup that are removed because the lower-level script already has the same functionality. This PR also includes updates to use the new fix organization. This includes the addition of a fix versions file, updates to the link script, and some changes in the fix directories used by scripts to account for files that have been relocated. The versions file sets the version number for each component of fix, so that fix can be more easily maintained and documented. The initial versions are all the same, and correspond to the old fix_NEW directory (other than some directories have been renamed or reorganizated). Finally, the link script has been updated to match the syntax of the rest of the build system (`checkout.sh` and `build_all.sh`). `link_workflow.sh` now detects the machine automatically instead of requiring an argument, and dev mode, which used to be set using `emc` as an argument, is now assumed. To run in ops mode (copy instead of link), the `-o` option is used. The full syntax is now simply: ``` ./link_workflow.sh [-o] -o: Run in operations (NCO) mode (copy instead of link) ``` This commit uses a temporary clone of the UFS_UTILS repo pending the merger of ufs-community/UFS_UTILS#688. Fixes NOAA-EMC#356
WalterKolczynski-NOAA
added a commit
to WalterKolczynski-NOAA/global-workflow
that referenced
this pull request
Aug 31, 2022
Removes all of the code and scripts that were moved to the new gfs-utils repo and adjusts workflow scripts to build and use them from the new location. Some of the build scripts had unnecessary calls to machine-setup that are removed because the lower-level script already has the same functionality. This PR also includes updates to use the new fix organization. This includes the addition of a fix versions file, updates to the link script, and some changes in the fix directories used by scripts to account for files that have been relocated. The versions file sets the version number for each component of fix, so that fix can be more easily maintained and documented. The initial versions are all the same, and correspond to the old fix_NEW directory (other than some directories have been renamed or reorganizated). Finally, the link script has been updated to match the syntax of the rest of the build system (`checkout.sh` and `build_all.sh`). `link_workflow.sh` now detects the machine automatically instead of requiring an argument, and dev mode, which used to be set using `emc` as an argument, is now assumed. To run in ops mode (copy instead of link), the `-o` option is used. The full syntax is now simply: ``` ./link_workflow.sh [-o] -o: Run in operations (NCO) mode (copy instead of link) ``` This commit uses a temporary clone of the UFS_UTILS repo pending the merger of ufs-community/UFS_UTILS#688. Fixes NOAA-EMC#356
Good catch. |
|
The |
GeorgeGayno-NOAA
requested changes
Sep 1, 2022
WalterKolczynski-NOAA
added a commit
to WalterKolczynski-NOAA/UFS_UTILS
that referenced
this pull request
Sep 1, 2022
Some additional scripts needed to be updated for the fix reorganization. A few files still contain paths remincent of the old paths, either to places the data is hosted on the web (FTP or AWS), or hard-coded locations not under global-workflow control. Refs: ufs-community#688
|
Updated a bunch more paths that I missed the first time. The remaining paths weren't changed because they point to a static path, either for FTP/AWS, or a personal location not controlled by workflow. |
WalterKolczynski-NOAA
force-pushed
the
feature/update_fix
branch
from
16c7316 to
b28f0f0
Compare
WalterKolczynski-NOAA
added a commit
to WalterKolczynski-NOAA/UFS_UTILS
that referenced
this pull request
Sep 6, 2022
Some additional scripts needed to be updated for the fix reorganization. A few files still contain paths remincent of the old paths, either to places the data is hosted on the web (FTP or AWS), or hard-coded locations not under global-workflow control. Refs: ufs-community#688
WalterKolczynski-NOAA
force-pushed
the
feature/update_fix
branch
from
b28f0f0 to
517c242
Compare
|
@GeorgeGayno-NOAA I also have a couple local changes I made to do my testing, I don't know if you wanted me to add them to this PR:
Should I:
|
Let's add them under a separate issue/PR. I tried to create a mpiexec.wcoss2 file one afternoon. I could not get it to work. Thanks for doing this. |
Opened two new issues (#690 and #691). I already have the fixes, I just need to sort them into PRs. |
JeffBeck-NOAA
approved these changes
Sep 6, 2022
|
@WalterKolczynski-NOAA The chgres_cube baselines store the surface fix files in
to each of the chgres_cube test scripts (e.g. c96.gfs.sigio.sh). Do you think this should happen or that the baseline surface fix directories should be renamed on each HPC? |
|
@DavidHuber-NOAA I would recommend doing what we did when we made this change to fix: move the baseline to the new locations and temporarily set up symlinks pointing the old locations to the new, so older versions don't break. |
|
Also, are these different tests than I get when I run |
|
@WalterKolczynski-NOAA OK, that sounds reasonable to me regarding the links, though I would defer to @GeorgeGayno-NOAA. The regression tests are run via rt.sh. That script needs to be modified first at the top by specifying a working directory, account, etc. Running |
Only the chgres_cube regression tests are failing. If you make the fix to ./ush/chgres_cube.sh (see my comment) I can run them for you. |
Which comment? I think I've already made the updates from this comment. I think the remaining issues lie in the baseline location, which is what David and I were talking about yesterday. |
|
Is there anything you are waiting on from me that is holding this up? |
Merge in the latest updates from 'develop' to your branch. And did you address the problem with the chgres_cube regression test script? See my review. |
|
I can run the chgres test for you after you update the script. |
Global workflow recently updated their fix directory structure to incorporate versioning using dates within the directory tree. Additionally, some files were moved and directories renamed (including removing all the 'fix_' prefixes of subdirectories). This commit updates the fix paths in UFS utils to match the new directory structure, and also take version numbers for each of the fix component directories, similar to how versions are used for modulefiles.) Fixes ufs-community#687 Refs: NOAA-EMC/global-workflow#966
Some additional scripts needed to be updated for the fix reorganization. A few files still contain paths remincent of the old paths, either to places the data is hosted on the web (FTP or AWS), or hard-coded locations not under global-workflow control. Refs: ufs-community#688
WalterKolczynski-NOAA
force-pushed
the
feature/update_fix
branch
from
517c242 to
9bf2ddb
Compare
|
I updated the branch. I think the issue your talking about are resolve, but see my earlier comment asking which comment you are referring to and the discussion with David about whether the baseline fix files need to be changed to match. |
|
@WalterKolczynski-NOAA Please make this change: |
|
@WalterKolczynski-NOAA And the jet directories need to be updated: |
|
@GeorgeGayno-NOAA Changes made. Unit tests and RT pass on Hera after the changes. |
GeorgeGayno-NOAA
approved these changes
Sep 20, 2022
There was a problem hiding this comment.
All regression tests now passing on all machines. Thanks, @WalterKolczynski-NOAA
WalterKolczynski-NOAA
added a commit
to WalterKolczynski-NOAA/global-workflow
that referenced
this pull request
Oct 4, 2022
Removes all of the code and scripts that were moved to the new gfs-utils repo and adjusts workflow scripts to build and use them from the new location. Some of the build scripts had unnecessary calls to machine-setup that are removed because the lower-level script already has the same functionality. This PR also includes updates to use the new fix organization. This includes the addition of a fix versions file, updates to the link script, and some changes in the fix directories used by scripts to account for files that have been relocated. The versions file sets the version number for each component of fix, so that fix can be more easily maintained and documented. The initial versions are all the same, and correspond to the old fix_NEW directory (other than some directories have been renamed or reorganizated). Finally, the link script has been updated to match the syntax of the rest of the build system (`checkout.sh` and `build_all.sh`). `link_workflow.sh` now detects the machine automatically instead of requiring an argument, and dev mode, which used to be set using `emc` as an argument, is now assumed. To run in ops mode (copy instead of link), the `-o` option is used. The full syntax is now simply: ``` ./link_workflow.sh [-o] -o: Run in operations (NCO) mode (copy instead of link) ``` This commit uses a temporary clone of the UFS_UTILS repo pending the merger of ufs-community/UFS_UTILS#688. Fixes NOAA-EMC#356
AnnetteGibbs-NOAA
added a commit
to AnnetteGibbs-NOAA/UFS_UTILS
that referenced
this pull request
Jul 3, 2023
* origin/develop: Update ccpp submodule (ufs-community#827) Change length of character variable "mosaic_name" in "make_solo_mosaic.c" (ufs-community#824) Update GDAS INIT utility for GFS COM reorganization (ufs-community#820) Add backup calculation for orography and mask generation (ufs-community#713) Improve repository build script (ufs-community#819) Use latest wgrib2 executable on Jet. (ufs-community#816) chgres_cube - Minor update for high-resolution grids . (ufs-community#814) Update PR template (ufs-community#813) global_cycle - Add soil moisture nudging for NoahMP option (ufs-community#809) Release v1.10.0 (ufs-community#807) Run sfc_climo_gen utility on Hera (ufs-community#791) Fix bug in chgres_cube subroutine search_many (ufs-community#808) sfc_climo_gen - Output fractions of each vegetation/soil type category (ufs-community#748) Replace w3nco library with w3emc (ufs-community#802) Fix Debug compilation (ufs-community#800) Update WCOSS2 build module to use new ESMF v8.4.1 library. (ufs-community#799) Update chgres_cube documentation for duplicate grib2 records (ufs-community#795) Use latest Cray modules on WCOSS2 (ufs-community#792) Remove checksum attribute from global_cycle restart files (ufs-community#794) Remove goto statements from chgres_cube (ufs-community#775) FVCOM_TOOLS - Remove dependency on module_nwp_base.f90 (ufs-community#790) Remove "gcovr" step from the 'developer' workflow (ufs-community#785) Update build module for Cheyenne (ufs-community#783) GDAS Init utility - consolidate the copy of coldstart files (ufs-community#773) Remove support for ODIN machine (ufs-community#782) Update GDAS INIT utility to optionally use fracoro data (ufs-community#741) Option to build only application specific utilities (e.g. GFS) (ufs-community#777) Use combined IP/IP2 library (ufs-community#695) Initial updates to global_cycle for Noah-MP land model (ufs-community#774) New resolution options for the cpld_gridgen utility (ufs-community#769) Use new EPIC-maintained hpc-stack on Jet (ufs-community#771) Update GDAS initialization scripts for the new ENKF directory (ufs-community#764) Reduce memory usage in chgres_cube (ufs-community#766) Detect duplicate vertical levels in chgres_cube. (ufs-community#767) Point to new input orography directory. (ufs-community#758) Release version 1.9 (ufs-community#754) HAFSv1 grid nesting updates (ufs-community#752) Remove compiler warnings from chgres_cube (ufs-community#747) chgres_cube: Split input_data module into to 3 separate modules (ufs-community#744) More work on CI, checking with different versions of ESMF (ufs-community#742) Add processing of soil color to sfc_climo_gen (ufs-community#721) more work on CI - added Linux_versions workflow (ufs-community#739) Add TEST_FILE_DIR option to CMake build, where test data files can be found instead of using FTP. (ufs-community#732) adding developer workflow (ufs-community#724) Update GDAS INIT utility for v16.3. (ufs-community#723) OROG_GSL - Remove negative bias in orographic asymmetery (OA) fields (ufs-community#718) Incorporate BNU soil type data (ufs-community#717) High-resolution MODIS and STATSGO veg/soil data. (ufs-community#703) Run WCOSS2 consistency tests under role account (ufs-community#711) Fix macOS CI workflow (ufs-community#715) Add utility codes to create BNU soil texture data (ufs-community#707) Update chgres_cube to output netcdf4 file. (ufs-community#704) Incorporate high-res global VIIRS vegetation data. (ufs-community#699) Run grid_gen consistency tests in parallel. (ufs-community#697) Fix error handling in "link_fixdirs.sh". Add -L to the copy command. (ufs-community#701) Update for new fixed data directory structure (ufs-community#688) Add WCOSS2 support for tests (ufs-community#693) Increase test data pull timeout (ufs-community#692) Option to install binaries to any directory. (ufs-community#685) Update global_cycle to use the latest CCPP version of sfcsub.F (ufs-community#671) Incorporate weight_gen program. (ufs-community#677) Add install of jpeg-turbo to macos workflows (ufs-community#684) Improve logic in regression test driver script (ufs-community#681) Release v1.8 (ufs-community#679) Update link_fixdirs.sh for new data directories (ufs-community#672) Update default ice climatology in ./ush/global_cycle.sh. (ufs-community#664) Remove all references to WCOSS 1 from UFS_UTILS (ufs-community#667) Update workflows to use latest macOS and ubuntu (ufs-community#675) Update GDAS Initialization utility for recent HPSS tarball name change (ufs-community#666) Port UFS_UTILS to WCOSS2 (ufs-community#642) Fix chgres_cube to process GEFS GRIB2 data (ufs-community#658) Update global_cycle_driver.sh for GFS OPS directory convention (ufs-community#655) Update documentation for SRW App (ufs-community#656) New coupled model utility (ufs-community#647) Update support for S4 and enable regression testing (ufs-community#654) global_cycle - Link to CCPP version of sfcsub.F (ufs-community#636) Add processing of new global AFWA snow data to emcsfc_snow2mdl. (ufs-community#648) Update build module on Cheyenne (ufs-community#646) Move to Intel 2022 on Jet, Hera and Orion (ufs-community#650) Host doxygen documentation for multiple releases (ufs-community#644) Download unit test data as part of the CMake build (ufs-community#630) chgres_cube - Complete removal of wgrib2 library (ufs-community#641) Eliminate circular dependency in chgres_cube Update workflow files to pull netcdf-c library from GitHub chgres_cube - Remove the wgrib2 library from the GRIB2 data read routines. Undefined symbols on macOS with Intel compiler (ufs-community#628) Update FVCOM code to handle sub-domain restart files using multiple cores. (ufs-community#624) chgres_cube - Run routine 'convert_omega' on all tasks. (ufs-community#627) Use ESMF 8.2.0 library Automate update of consistency test baseline data. (ufs-community#603) Update workflow files to use newer versions of ESMF and NCEPLIBS. (ufs-community#617) Update build modules to be lua compliant (ufs-community#614) Allow FVCOM tools to Update Ice Surface Roughness Length (ufs-community#604) Update the requested memory in the Orion chgres_cube consistency test script (ufs-community#611) Use copy of grib_util under Jet role account. (ufs-community#608) Run consistency tests on Orion using role account (ufs-community#606) Run consistency tests on Hera using role account (ufs-community#605) Run consistency tests on Jet using role account. (ufs-community#607) Update more documentation after move to ufs-community (ufs-community#597) fvcom_tools - Add option to process 'cold' or 'warm' restart files (ufs-community#595) Update documentation after move to ufs-community (ufs-community#594) chgres_cube - Eliminate segmentation fault in input_data.F90 (ufs-community#585) Update to language of unit test README to match that in unit test. chgres_cube - Simplify surface processing using field bundles (ufs-community#572) Add compiler flags for GNU Fortran v10 or newer compilers. (ufs-community#583) Move verbose output from example unit test to be commented to streamline test output. Updates to test README to add instructions for use of example unit test. Add Findwgrib2.cmake (ufs-community#578) Added unit test to be used as an instructional example for new users.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
Global workflow recently updated their fix directory structure to incorporate versioning using dates within the directory tree. Additionally, some files were moved and directories renamed (including removing all the 'fix_' prefixes of subdirectories). This commit updates the fix paths in UFS utils to match the new directory structure, and also take version numbers for each of the fix component directories, similar to how versions are used for modulefiles.)
TESTS CONDUCTED:
No new unit tests required.
DEPENDENCIES:
None
DOCUMENTATION:
None needed
ISSUE:
Fixes #687
Refs: NOAA-EMC/global-workflow#966
CONTRIBUTORS (optional):
@KateFriedman-NOAA did a lot of the work before I finished it off and tested.