Partial automation for Unified environment testing of zppy

[forsyth2]

This script provides partial automation for testing zppy in Unified Release Candidate (RC) environments:

# https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html

# BEFORE RUNNING ################################################

# Set these values:
version_num=3 # What attempt of the test script is this?

unified_rc=1.10.0rc1
machine=chrysalis

conda_path=/home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh
zppy_dir=~/zppy/
e3sm_diags_dir=~/e3sm_diags/
output_dir=/lcrc/group/e3sm/ac.forsyth2/

# cd into zppy_dir and e3sm_diags_dir
# In each, run \`git status\` to make sure there's no changes
# that may accidentally get written over.

# BEGIN AUTOMATIC PORTION OF SCRIPT #############################
unique_id=${unified_rc}v${version_num}

# Step 2
cd ${zppy_dir}

# Step 3b
git fetch upstream main
git checkout -b test_unified_${unique_id}_${machine} upstream/main
if [ $? != 0 ]; then
  echo "ERROR (1): Could not checkout fresh zppy branch"
  exit 1
fi

# Step 4
cd ${e3sm_diags_dir}
git checkout main
if [ $? != 0 ]; then
  echo "ERROR (2): Could not checkout e3sm_diags main"
  exit 2
fi
git fetch upstream
git reset --hard upstream/main
source ${conda_path}
mamba clean --all
e3sm_diags_env=e3sm_diags_test_${unique_id}
mamba env create -f conda-env/dev.yml -n ${e3sm_diags_env}
conda activate ${e3sm_diags_env}
pip install .
cd ${zppy_dir}

# Step 6
if [[ ${machine} == "chrysalis" ]]; then
  unified_rc_env=/lcrc/soft/climate/e3sm-unified/test_e3sm_unified_${unified_rc}_chrysalis.sh
elif [[ ${machine} == "compy" ]]; then
  unified_rc_env=/share/apps/E3SM/conda_envs/test_e3sm_unified_${unified_rc}_compy.sh
elif [[ ${machine} == "pm-cpu" ]]; then
  unified_rc_env=/global/common/software/e3sm/anaconda_envs/test_e3sm_unified_${unified_rc}_pm-cpu.sh
else
  echo "ERROR (3): Could not determine unified_rc_env"
fi
source ${unified_rc_env}

# Step 7
echo "Running unit tests:"
python -u -m unittest tests/test_*.py

# Begin instructions to the developer ####################################################################
echo "Confirm the unit tests above passed"
cd ${e3sm_diags_dir}
echo "Make sure this commit matches the latest at https://github.com/E3SM-Project/e3sm_diags/commits/main"
git log | head -n 5
cd ${zppy_dir}
echo "Make sure this commit matches the latest at https://github.com/E3SM-Project/zppy/commits/main"
git log | head -n 5

# (Step 5)
echo "Edit ${zppy_dir}/tests/integration/utils.py as follows:"
echo "1. Set \`UNIQUE_ID\` to ${unique_id}"
echo "2. Under \`get_${machine}_expansions\`, set \`diags_environment_commands\` to be \`source ${conda_path}; conda activate ${e3sm_diags_env}\`"
echo "3. Under \`get_${machine}_expansions\`, set \`environment_commands_test\` to be \`source ${unified_rc_env}\`"
echo "4. Change the last line \`generate_cfgs()\` to \`generate_cfgs(unified_testing=True)\`"

# (Step 8)
echo "Run \`python tests/integration/utils.py\`"
echo "Run \`zppy -c tests/integration/generated/test_complete_run_${machine}.cfg\`"
echo "Run \`zppy -c tests/integration/generated/test_bundles_${machine}.cfg\`"
echo "Wait for the last command to finish and then re-run:"
echo "Run \`zppy -c tests/integration/generated/test_bundles_chrysalis.cfg\`"
echo "Wait for all of these commands to finish"
echo "Now check that no errors were generated by running:"
echo "cd ${output_dir}zppy_test_complete_run_output/${unique_id}/v2.LR.historical_0201/post/scripts; grep -v 'OK' *.status; cd-"
echo "cd ${output_dir}zppy_test_bundles_output/${unique_id}/v2.LR.historical_0201/post/scripts; grep -v 'OK' *.status; cd-"
echo "Now actually run the integration tests:"
echo "Run \`python -u -m unittest tests/integration/test_*.py\`"

There are some steps we just need the tester to run themselves manually since they involve checking on values or waiting for launched jobs to finish. The script at least provides exact instructions on what to do. For example:

Confirm the unit tests above passed
Make sure this commit matches the latest at https://github.com/E3SM-Project/e3sm_diags/commits/main
commit e7ed2deb750143d0726ccbd729e6c7635ee240b4
Author: Jill Chengzhu Zhang <[email protected]>
Date:   Sun Apr 14 15:18:05 2024 -0700

    update for 2.12.0rc1 (#807)
Make sure this commit matches the latest at https://github.com/E3SM-Project/zppy/commits/main
commit 0dfa9e66e5475ed1f1f39bda1c6d5a4ac9d266f3
Author: forsyth2 <[email protected]>
Date:   Fri Apr 12 17:25:06 2024 -0700

    Bump to 2.5.0rc1 (#577)
Edit /home/ac.forsyth2/zppy//tests/integration/utils.py as follows:
1. Set `UNIQUE_ID` to 1.10.0rc1v3
2. Under `get_chrysalis_expansions`, set `diags_environment_commands` to be `source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_test_1.10.0rc1v3`
3. Under `get_chrysalis_expansions`, set `environment_commands_test` to be `source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.10.0rc1_chrysalis.sh`
4. Change the last line `generate_cfgs()` to `generate_cfgs(unified_testing=True)`
Run `python tests/integration/utils.py`
Run `zppy -c tests/integration/generated/test_complete_run_chrysalis.cfg`
Run `zppy -c tests/integration/generated/test_bundles_chrysalis.cfg`
Wait for the last command to finish and then re-run:
Run `zppy -c tests/integration/generated/test_bundles_chrysalis.cfg`
Wait for all of these commands to finish
Now check that no errors were generated by running:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/1.10.0rc1v3/v2.LR.historical_0201/post/scripts; grep -v 'OK' *.status; cd-
cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_bundles_output/1.10.0rc1v3/v2.LR.historical_0201/post/scripts; grep -v 'OK' *.status; cd-
Now actually run the integration tests:
Run `python -u -m unittest tests/integration/test_*.py`

[forsyth2]

This is a very partial solution to #516 (part of #553, which in turn is part of #572)