Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MACOS and Generic Linux options for regional_workflow #402

Merged
merged 50 commits into from
Feb 1, 2021
Merged

Add MACOS and Generic Linux options for regional_workflow #402

Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
9340c97
Adapt new bash syntax to be more back-compatible with older and more …
mkavulich Nov 18, 2020
ee7e914
Add if-blocks where necessary to call "greadlink -f" on Darwin/MacOS …
mkavulich Nov 18, 2020
95e2cfc
Add new bash_utils file "define_macos_utilities.sh" which defines var…
mkavulich Nov 18, 2020
df16952
Finally get rid of the spurious and misleading error messages at the …
mkavulich Nov 18, 2020
39294b4
Check for special MacOS equivalent utilities, abort if not found
mkavulich Nov 18, 2020
6fcda9c
Add more helpful error messages for missing `greadlink` command on Da…
mkavulich Nov 18, 2020
2935fa3
Simplify greadlink/readlink if blocks to variable $READLINK where pos…
mkavulich Nov 18, 2020
5350613
Re-work logic of hard-coded IC and LBC directories. Now for any platf…
mkavulich Nov 19, 2020
1760c89
Since data may also exist on HPSS, only fail in set_extrn_mdl_params.…
mkavulich Nov 20, 2020
03f8c9a
Better error messages if there is no way to get input data
mkavulich Nov 20, 2020
c0ea426
Add WORKFLOW_MANAGER as a variable (currently set to "rocoto" for all…
mkavulich Nov 23, 2020
ad541e9
date utility needs a macos-specific gdate as well
mkavulich Nov 23, 2020
81b20f8
ln also needs to use macos-specific gln (BSD version does not accept …
mkavulich Nov 23, 2020
927675f
Another macos bash fix: the "-v" test is only available for bash 4.2 …
mkavulich Nov 23, 2020
121df9a
Bad practice to grep piped output from grep (causes errors on MacOS b…
mkavulich Nov 23, 2020
857c986
In get_extrn_mdl_file_dir_info.sh, don't fail on check for pre-staged…
mkavulich Nov 23, 2020
0f4f453
ulimit -s unlimited doesn't work without admin privileges on macos. R…
mkavulich Nov 24, 2020
14f51b2
Need to create the logdir as well, otherwise later touch commands may…
mkavulich Nov 24, 2020
4700266
Forgot to add LN_UTIL for non-MACOS platforms
mkavulich Dec 3, 2020
6ed0afc
Need to initialize "sysdir" to a blank value so workflow generation d…
mkavulich Dec 3, 2020
a812ce3
Add stanzas for generic linux ("LINUX") for generation script
mkavulich Dec 3, 2020
32eb0f9
Add LINUX stanzas for make_grid and make_orog tasks, remove unnecessa…
mkavulich Dec 3, 2020
c6ff070
Fix mistake for bash portability at run_fcst step
mkavulich Jan 13, 2021
8bec4dd
Add check for ncdump in environment for more controlled failures
mkavulich Jan 13, 2021
01792f6
Add mpirun commands for LINUX (will make configurable later), add cas…
mkavulich Jan 13, 2021
47e8750
FINALLY got the generic linux working.
mkavulich Jan 17, 2021
6e4be94
Allow run commands to be set in config.sh for LINUX and MACOS
mkavulich Jan 17, 2021
bce8f4c
A couple fixes to my cross-platform fixes
mkavulich Jan 17, 2021
f248b4d
More fixes to my fixes
mkavulich Jan 19, 2021
bf3ad0e
Another fix
mkavulich Jan 19, 2021
713699f
Better way of setting default run_fcst command: use the calculated nu…
mkavulich Jan 20, 2021
9071ba3
Replace backticks with more modern and less troublesome parenthetical…
mkavulich Jan 25, 2021
194f565
Fix copy-paste errors that removed "local" designation from some vari…
mkavulich Jan 25, 2021
6ace3c9
Merge branch 'release/public-v1' into add_macos_generic_linux_workflow
mkavulich Jan 26, 2021
5430c63
Fix more accidental variable globalizations, tidy up some whitespace …
mkavulich Jan 26, 2021
df3026b
Simplify Darwin-specific bash utility errors
mkavulich Jan 26, 2021
0bc56d4
Fix call to $DATE_UTIL
mkavulich Jan 27, 2021
d17b2e3
Forgot to modify DATE_UTIL calls in exregional_run_post
mkavulich Jan 27, 2021
4838b4b
Some suggestions/fixes from Gerard
mkavulich Jan 29, 2021
2da62fd
Remove files suggested as unneeded and outdated by Gerard
mkavulich Jan 29, 2021
9487bbb
Create functions to convert strings to all lowercase or uppercase to …
mkavulich Jan 29, 2021
6959f45
Fix a few more accidental non-local variables
mkavulich Jan 29, 2021
6b3c5f4
With new "echo_lowercase" function these temp variables are no longer…
mkavulich Jan 29, 2021
1761db2
Replace all remaining sed commands with $SED for consistency's sake
mkavulich Jan 29, 2021
f81781c
Forgot $SED is not defined here yet
mkavulich Jan 29, 2021
46654e7
One more fix from Gerard
mkavulich Jan 29, 2021
7a22549
Fix dumb mistake in launch script
mkavulich Jan 30, 2021
cc45725
Add requested clarifying comment
mkavulich Jan 30, 2021
18b01ed
Merge branch 'release/public-v1' into add_macos_generic_linux_workflow
mkavulich Jan 30, 2021
856076e
Forgot some final "DATE_UTIL" instances, these seem harmless but shou…
mkavulich Jan 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_GET_EXTRN_MDL_FILES
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_GRID
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_ICS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_LBCS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_OROG
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_SFC_CLIMO
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_RUN_FCST
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down
4 changes: 2 additions & 2 deletions jobs/JREGIONAL_RUN_POST
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -100,7 +100,7 @@ cd_vrfy "${fhr_dir}"
#
#-----------------------------------------------------------------------
#
fhr=$( printf "%s" "${fhr}" | sed -n -r -e "s/^([0-9]+)$/\1/p" )
fhr=$( printf "%s" "${fhr}" | $SED -n -r -e "s/^([0-9]+)$/\1/p" )
if [ -z "$fhr" ]; then
print_err_msg_exit "\
The forecast hour (fhr) must be a non-empty string consisting of only
Expand Down
6 changes: 3 additions & 3 deletions scripts/exregional_get_extrn_mdl_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -524,7 +524,7 @@ not happen."
# to the current directory. Then move the files.
#
rel_dir=$( printf "%s" "${extrn_mdl_arcvrel_dir}" | \
sed -r 's%^(\/|\.\/)([^/]*)(.*)%\2\3%' )
$SED -r 's%^(\/|\.\/)([^/]*)(.*)%\2\3%' )
mv_vrfy ${rel_dir}/* .
#
# Get the first subdirectory in rel_dir, i.e. the subdirectory before the
Expand All @@ -533,7 +533,7 @@ not happen."
# it.
#
subdir_to_remove=$( printf "%s" "${rel_dir}" | \
sed -r 's%^([^/]*)(.*)%\1%' )
$SED -r 's%^([^/]*)(.*)%\1%' )
rm_vrfy -rf ./${subdir_to_remove}
#
# If extrn_mdl_arcvrel_dir does not start with a "/" (and it is not
Expand Down
20 changes: 18 additions & 2 deletions scripts/exregional_make_grid.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -147,6 +147,14 @@ case $MACHINE in
ulimit -a
;;

"MACOS")
APRUN=time
;;

"LINUX")
APRUN=time
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
Expand Down Expand Up @@ -433,11 +441,19 @@ uniform cubed-sphere grid equivalent resolution returned with nonzero exit
code:
exec_fp = \"${exec_fp}\""

# Make sure 'ncdump' is available before we try to use it
if ! command -v ncdump &> /dev/null
then
print_err_msg_exit "\
The utility 'ncdump' was not found in the environment. Be sure to add the
netCDF 'bin/' directory to your PATH."
fi

# Make the following (reading of res_equiv) a function in another file
# so that it can be used both here and in the exregional_make_orog.sh
# script.
res_equiv=$( ncdump -h "${grid_fp}" | \
grep -o ":RES_equiv = [0-9]\+" | grep -o "[0-9]" ) || \
grep -o ":RES_equiv = [0-9]\+" | sed 's/[^0-9]*//g' ) || \
mkavulich marked this conversation as resolved.
Show resolved Hide resolved
print_err_msg_exit "\
Attempt to extract the equivalent global uniform cubed-sphere grid reso-
lution from the grid file (grid_fp) failed:
Expand Down
17 changes: 16 additions & 1 deletion scripts/exregional_make_ics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -116,6 +116,21 @@ case "$MACHINE" in
APRUN="ibrun"
;;

"MACOS")
APRUN=$RUN_CMD_UTILS
;;

"LINUX")
APRUN=$RUN_CMD_UTILS
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
MACHINE = \"$MACHINE\"
APRUN = \"$APRUN\""
;;

esac
#
#-----------------------------------------------------------------------
Expand Down
17 changes: 16 additions & 1 deletion scripts/exregional_make_lbcs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -116,6 +116,21 @@ case "$MACHINE" in
APRUN="ibrun"
;;

"MACOS")
APRUN=$RUN_CMD_UTILS
;;

"LINUX")
APRUN=$RUN_CMD_UTILS
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
MACHINE = \"$MACHINE\"
APRUN = \"$APRUN\""
;;

esac
#
#-----------------------------------------------------------------------
Expand Down
10 changes: 9 additions & 1 deletion scripts/exregional_make_orog.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -147,6 +147,14 @@ case $MACHINE in
export APRUN="time"
;;

"MACOS")
APRUN=time
;;

"LINUX")
APRUN=time
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
Expand Down
10 changes: 9 additions & 1 deletion scripts/exregional_make_sfc_climo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -170,6 +170,14 @@ case $MACHINE in
APRUN="ibrun -np ${nprocs}"
;;

"MACOS")
APRUN=$RUN_CMD_UTILS
;;

"LINUX")
APRUN=$RUN_CMD_UTILS
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
Expand Down
14 changes: 11 additions & 3 deletions scripts/exregional_run_fcst.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -144,6 +144,14 @@ case $MACHINE in
APRUN="ibrun -np ${PE_MEMBER01}"
;;

"MACOS")
APRUN=$RUN_CMD_FCST
;;

"LINUX")
APRUN=$RUN_CMD_FCST
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
Expand Down Expand Up @@ -360,9 +368,9 @@ for (( i=0; i<${num_symlinks}; i++ )); do

mapping="${CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING[$i]}"
symlink=$( printf "%s\n" "$mapping" | \
sed -n -r -e "s/${regex_search}/\1/p" )
$SED -n -r -e "s/${regex_search}/\1/p" )
target=$( printf "%s\n" "$mapping" | \
sed -n -r -e "s/${regex_search}/\2/p" )
$SED -n -r -e "s/${regex_search}/\2/p" )

symlink="${run_dir}/$symlink"
target="$FIXam/$target"
Expand Down
14 changes: 11 additions & 3 deletions scripts/exregional_run_post.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
Expand Down Expand Up @@ -135,6 +135,14 @@ case $MACHINE in
APRUN="ibrun -n $nprocs"
;;

"MACOS")
APRUN=$RUN_CMD_POST
;;

"LINUX")
APRUN=$RUN_CMD_POST
;;

*)
print_err_msg_exit "\
Run command has not been specified for this machine:
Expand Down Expand Up @@ -207,7 +215,7 @@ tmmark="tm00"
dyn_file="${run_dir}/dynf${fhr}.nc"
phy_file="${run_dir}/phyf${fhr}.nc"

post_time=$( date --utc --date "${yyyymmdd} ${hh} UTC + ${fhr} hours" "+%Y%m%d%H" )
post_time=$( $DATE_UTIL --utc --date "${yyyymmdd} ${hh} UTC + ${fhr} hours" "+%Y%m%d%H" )
post_yyyy=${post_time:0:4}
post_mm=${post_time:4:2}
post_dd=${post_time:6:2}
Expand Down Expand Up @@ -279,7 +287,7 @@ mv_vrfy BGRD3D.GrbF${post_fhr} ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}.${tm
# space inserted between the dd and hh. If so, just use "$yyyymmdd $hh"
# instead of calling sed.
start_date=$( echo "${cdate}" | sed 's/\([[:digit:]]\{2\}\)$/ \1/' )
basetime=$( date +%y%j%H%M -d "${start_date}" )
basetime=$( $DATE_UTIL +%y%j%H%M -d "${start_date}" )
ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgdawpf${fhr}.${tmmark}.grib2 \
${postprd_dir}/BGDAWP_${basetime}f${fhr}00
ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}.${tmmark}.grib2 \
Expand Down
Loading