Replies: 8 comments 23 replies
-
Friday 2024-10-18Note that before we get to testing a release candidate for Unified, we'll need to get v3 test data on to Perlmutter and Compy too. On Chrysalis: # Set up branch
cd ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241018 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/15: make --closedcontourcmd resolution specific (#824)
mamba clean --all
# Press `y` twice
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241018
conda activate e3sm_diags_20241018
pip install .
# Set up testing files
cd ../zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241018"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241018",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241018
conda activate zppy_dev_weekly_20241018
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241018/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Run integration tests
python -u -m unittest tests/integration/test_*.pyThis gets us these results: We can see the diff is from the formatting change introduced in #627, so this is expected. Let's update the expected results. That gives: ✅ This week's testing of |
Beta Was this translation helpful? Give feedback.
0 replies
-
Friday 2024-10-25Commits since 2024-10-18:
# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241025 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/23: Fixing African easterly wave density plots in TC analysis (#851)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241025
conda activate e3sm_diags_20241025
pip install .
# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241025"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241025 # This command took a while
conda activate zppy_dev_weekly_20241025
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241025
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o614397
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/lnd/180x360_aave/clim/2yr/v3.LR.historical_0051_*_1987??_1988??_climo.nc': No such file or directory
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o614424
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/lnd/180x360_aave/clim/2yr/v2.LR.historical_0201_*_1982??_1983??_climo.nc': No such file or directory
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.pyThat gives: The The Let's look at the image check failures: Let's try the directory paths: Clearly, image check failures aren't being routed to the right place. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Bug report is at #635. |
Beta Was this translation helpful? Give feedback.
-
Friday 2024-11-01Commits since 2024-10-25:
# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241101 upstream/main
git log
# Last commit, from 10/30: Updates to diags handling (#633)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241101
conda activate e3sm_diags_20241101
pip install .
# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241101"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241101 # This command took a while
conda activate zppy_dev_weekly_20241101
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 23 tests pass (increased from 22)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241101
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241101/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.pyThat gives: The diff is from the template file change in https://github.com/E3SM-Project/zppy/pull/633/files#diff-fbd1002ca5f9034c321e896884792a69fc7e4a9fc14e826a3954dc81a3d6d76d, so this is expected. Let's update the expected results. That test passes now. As for the image check failures (attempted to format this hierarchically, but only semi-successfully):
So, all image check failures are accounted for. Let's update the expected results. After about 20 minutes, that gives: All expected results have been updated. ✅ This week's testing of |
Beta Was this translation helpful? Give feedback.
0 replies
-
Friday 2024-11-22This is the first Weekly test since the merging of major refactor splitting zppy into zppy itself (workflow coordination) and zppy-interfaces ("last-mile" code improving E3SM-specific usability of external packages). Check commit historyCommits since 2024-11-01 (last Weekly test of zppy:
However, I'm going to conciously test Set up branch# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: zppy-interfaces refactor (#642)Set up zppy-interfaces envcd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: Initial implementation (#1)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20241122
conda activate zi_dev_weekly_20241122
pip install .Set up e3sm_diags envcd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git checkout main
git reset --hard upstream/main
git log
# Last commit, from 11/20: Release 2.12.1 (#893)
# NOTE: As mentioned earlier, let's use the Diags code that was used for the last testing
git reset --hard 3f5b03627aef703b4f7d8463cd5faed3c0806566
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_from_20241029
conda activate e3sm_diags_from_20241029
pip install .Set up zppy envcd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20241122
conda activate zppy_dev_weekly_20241122
pip install .Unit tests for zppy-interfacescd ~/ez/zppy-interfaces
conda activate zi_dev_weekly_20241122
pytest tests/global_time_series/test_*.py
# 7 passed in 25.86sUnit tests for zppycd ~/ez/zppy
conda activate zppy_dev_weekly_20241122
pytest tests/test_*.py
# 23 passed in 0.63sIntegration testing for zppy# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241122"
# For get_chyrsalis_expansions:
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20241122",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241122
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029
# e3sm_to_cmip_environment_commands=
# environment_commands=
# Reminder: `e3sm_to_cmip_environment_commands=''` => the environment of the `ts` task will be used
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# NOTE: Created https://github.com/E3SM-Project/zppy/issues/643 to also print out `global_time_series_environment_commands`
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241122/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py
# This needs to be converted to pytest still -- see https://github.com/E3SM-Project/zppy/issues/644That gives: Errorsls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_defaults.py tests/integration/test_weekly.py
# tests/integration/test_campaign.py tests/integration/test_last_year.py
python -u -m unittest tests/integration/test_bash_generation.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_campaign.py
# FAILED (failures=5)
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_defaults.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_last_year.py
# OKFor Weekly tests, let's look at the image check failures
These issues should be simple to fix. Next week, I'll address them and then re-run the Weekly test suite. Specifically:
|
Beta Was this translation helpful? Give feedback.
1 reply
-
Rerun on branch
|
Beta Was this translation helpful? Give feedback.
-
|
Note that #520 is to automate the |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
NOTE: The weekly tests' expected results were updated as part of the merging of #650 today (2024-12-18). All tests pass. |
Beta Was this translation helpful? Give feedback.
0 replies
-
2025-01-14 testing for v3.0.0rc1# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20250114 upstream/main
git log
# Last commit, from 1/14: Update PR template (#658)
# Set up and test zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20250114 upstream/main
git log
# Last commit, from 1/14: Update PR template (#12)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20250114
conda activate zi_dev_weekly_20250114
pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 20.28s
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
# https://github.com/E3SM-Project/zppy/pull/651 (zppy using CDAT-migrated Diags) hasn't merged yet
# Because https://github.com/E3SM-Project/e3sm_diags/pull/907 (fixes for CDAT-migrated Diags) hasn't merged yet.
# So, we can't use the latest E3SM Diags.
# https://github.com/E3SM-Project/e3sm_diags/pull/902 (CDAT-migrated Diags) has 42 commits
# The first is:
# CDAT Migration Phase 2: Refactor core utilities and lat_lon set (#677) c090d34ae96dcf43159174ead5c7688385cc70fb
# From 2024-12-04
# The last commit before that was:
# Release 2.12.1 (#893) ca41b0e5d913610c88410928951f1ed11c75663f
# From 2024-11-20
git checkout -b diags_pre_cdat_migration ca41b0e5d913610c88410928951f1ed11c75663f
git log
# Confirmed last commit, from 11/20: Release 2.12.1 (#893)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_pre_cdat_migration
conda activate e3sm_diags_pre_cdat_migration
pip install .
# Set up zppy env
cd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20250114
conda activate zppy_dev_weekly_20250114
pip install .
# Unit tests for zppy
pytest tests/test_*.py
# 23 passed, 1 warning in 0.40s
# (DeprecationWarning: invalid escape sequence \.)
# Integration testing for zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20250114"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250114",
# Keep this as-is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20250114
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250114
# environment_commands=
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Wait to finish
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Wait to finish
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250114/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.30s
# Diffs are expected
pytest tests/integration/test_campaign.py
# 3 failed, 3 passed in 2.33s
# Diffs are expected
pytest tests/integration/test_defaults.py
# 1 failed in 1.25s
# Diffs are expected
pytest tests/integration/test_last_year.py
# 1 passed in 0.27s
pytest tests/integration/test_weekly.py
# FAILED tests/integration/test_weekly.py::test_comprehensive_v2_images - AssertionError
# FAILED tests/integration/test_weekly.py::test_comprehensive_v3_images - AssertionError
# 2 failed, 3 passed in 1290.08s (0:21:30)
# test_comprehensive_v2_images
# Total number of images checked: 4376
# Missing images: 2
# e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/qbo/QBO-model-vs-model/qbo_diags.png
# e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/qbo/QBO-ERA-Interim/qbo_diags.png
# test_comprehensive_v3_images
# Total number of images checked: 5216
# Missing images: 2
# e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/qbo/QBO-model-vs-model/qbo_diags.png
# e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/qbo/QBO-ERA-Interim/qbo_diags.png
# Mismatched images: 14
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_3.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_3.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_original.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_3.png
# Let's look at the image diffs
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/image_check_failures_comprehensive_v2/ is empty
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/ has diffs
# The diffs appear to be due to the unit changes introduced by zppy_interfaces/global_time_series/zppy_land_fields.csv
# What about the missing E3SM images though?
# QBO isn't showing up on:
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts
grep -in qbo e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758
# 1642:2025-01-14 15:38:56,458 [ERROR]: core_parameter.py(_run_diag:269) >> Error in e3sm_diags.driver.qbo_driver
# 1646: File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 261, in run_diag
# 1648: File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 170, in get_psd_from_wavelet
# Traceback (most recent call last):
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/parameter/core_parameter.py", line 266, in _run_diag
# single_result = module.run_diag(self)
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 261, in run_diag
# test["wave_period"], test_wavelet = get_psd_from_wavelet(test_detrended_data)
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 170, in get_psd_from_wavelet
# cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# AttributeError: module 'scipy.signal' has no attribute 'cwt'
grep -in cwt e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758:1649: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758:1650:AttributeError: module 'scipy.signal' has no attribute 'cwt'
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o665759:2297: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o665759:2298:AttributeError: module 'scipy.signal' has no attribute 'cwt'
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250114/v2.LR.historical_0201/post/scripts
grep -in cwt e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o665789:1790: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o665789:1791:AttributeError: module 'scipy.signal' has no attribute 'cwt'
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o665790:3483: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o665790:3484:AttributeError: module 'scipy.signal' has no attribute 'cwt' |
Beta Was this translation helpful? Give feedback.
18 replies
-
|
Ok latest testing appears to be working, so I'm going to updated expected results and get the Testing details# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git fetch xylar
git checkout -b test_zppy_weekly_20250116 upstream/main
git log
# Last commit, from 1/14: Update PR template (#658)
# Add Xylar's commit from https://github.com/E3SM-Project/zppy/pull/659
git cherry-pick 629289e8f532fc2f66d6c9ccb3637fb43179c8c9
git log
# Xylar's commit is there
# Set up and test zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20250116 upstream/main
git log
# Last commit, from 1/16: Bump to 0.1.1 (#13)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20250116
conda activate zi_dev_weekly_20250116
pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 22.99s
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git checkout diags_pre_cdat_migration_plus_constraint
# Branch created earlier
git log
# Confirm last 2 commits:
# 1/15: Add scipy constraint
# 11/20: Release 2.12.1 (#893)
# Use existing conda env
conda activate e3sm_diags_pre_cdat_migration_plus_constraint
pip install .
# Set up zppy env
cd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20250116
conda activate zppy_dev_weekly_20250116
pip install .
# Unit tests for zppy
pytest tests/test_*.py
# 23 passed in 0.23s
# Integration testing for zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20250116"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration_plus_constraint",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250116",
# Keep this as-is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20250116
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration_plus_constraint
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250116
# environment_commands=
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# Clear earlier dependent tasks
scancel -u ac.forsyth2
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Wait to finish
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Wait to finish
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250116/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250116/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20250116/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.44s
# Diffs are expected
pytest tests/integration/test_campaign.py
# 5 failed, 1 passed in 2.50s
# Diffs are expected
pytest tests/integration/test_defaults.py
# 1 failed in 1.01s
# Diffs are expected
pytest tests/integration/test_last_year.py
# 1 passed in 1.06s
pytest tests/integration/test_weekly.py
# FAILED tests/integration/test_weekly.py::test_comprehensive_v3_images - AssertionError
# 1 failed, 4 passed in 1293.95s (0:21:33)The errors are: Let's look at the image diffs. https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250116/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/ has diffs, which continue to appear to be due to the unit changes introduced by zppy_interfaces/global_time_series/zppy_land_fields.csv (or else minor changes from pixel shifting) |
Beta Was this translation helpful? Give feedback.
-
Updated expected results# Remembered to copy over the old expected results!
cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20250116
cd ~/ez/zppy
chmod u+x tests/integration/generated/update_*_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.49s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 failed, 5 passed in 1.74s
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mkdir -p /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mv test_campaign_high_res_v1_output/post/scripts/*.settings /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
pytest tests/integration/test_campaign.py
# 6 passed in 1.47s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.29s
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# 5 passed in 1298.62s (0:21:38) |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2, I've seen some more of these |
Beta Was this translation helpful? Give feedback.
-
|
I added MPAS-Dev/MPAS-Analysis#1052 to try to keep tack of it. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good, thanks @xylar |
Beta Was this translation helpful? Give feedback.
-
Thread for E3SM Unified 1.11.0 testingWhere is testing data currently?
2025-01-21source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc2_chrysalis.sh
conda list
conda list | grep -E "zppy-interfaces|zppy|zstash"
# zppy 3.0.0rc1 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# zstash 1.4.4rc1 pyh9caca29_0 conda-forge/label/zstash_devTo do:
|
Beta Was this translation helpful? Give feedback.
3 replies
-
If the test data lives in |
Beta Was this translation helpful? Give feedback.
-
|
@xylar the test data is unfortunately not in Chrysalis: cd /lcrc/group/e3sm/public_html/diagnostics
# 1.2T
# v3 test data
# I actually want to copy this to my directory, but that is time-consuming, so it continues to just exist here for now.
cd /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051
du -sh .
# 18T
# v2 test data
cd /lcrc/group/e3sm/ac.forsyth2/E3SMv2/v2.LR.historical_0201
du -sh .
# 24TSo, we're looking at 18+24=42T of data, compared to the roughly 1T in In all likelihood, we don't need to be storing so much data, but my main bottleneck on that is I'm not quite sure how to figure out which files are actually necessary and which go unused in the testing process. On Perlmutter: # v3 test data
cd /global/cfs/cdirs/e3sm/forsyth/E3SMv3/v3.LR.historical_0051
# I copied this data over from /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051 with `zstash create`
# using Globus, and then extracted with `zstash extract`. That process was quite lengthy -- at least a day and a half.
du -sh .
# 18 T
# v2 test data
cd /global/cfs/cdirs/e3sm/forsyth/E3SMv2/v2.LR.historical_0201
du -sh .
# 24 TAnother big problem with Perlmutter testing is the sheer length of time required to get node time. For Compy, the v2 data exists, but v3 data has not been copied over yet. These path prefixes are defined via |
Beta Was this translation helpful? Give feedback.
-
|
Oh, yeah, thanks for the reminder. That's a pretty enormous set of testing data to have to schlep around. It does seem worth figuring out whether there's a smaller subset you could use but that probably would take more time than just Globusing the data over to Compy. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Starting this page to log the process/results for the weekly runs of
zppy's integration testing: bundles, comprehensive_v2, comprehensive_v3Beta Was this translation helpful? Give feedback.
All reactions