Feature/questions to tasks dan (#205)

* fix a few questions * fixes for r2d2 * extensive override * bug fixes when adding string * remove extra model components key * clean up experiment yaml * Feature/jjin3/ufo tests x0048 assign error (#204) * add suite for obs filter tests * add ncdiag tasks * converter working (no geovals) * executable runs * sat bias correction conversion added * add aircraft yaml * move to R2D2 for getting the observations * convert to using GetObservations * append * uncomment * clean up hidden aspects of the config * move the opening of jedi interface files * add class to handle jedi file templating * stage tasks working * unify copyright date * pep8 * add utility for setting the list of tasks for each question * add question prompts * add more questions and defaults * move jedi exe base * refactor run jedi exe base * remove fgat as not needed * move depends * write lists the way python does * make sure jedi_forecast model is used * more q clean up * remove jedi exe base class from questions * convert task names to CamelCase in the questions * modify config class to create variables and methods to get them * Update generate_b_climatology_by_linking.py quick fix * common config availability * eva obs fix * use new variable functions for config instead of config_get * fix clean patterns * update question yaml generation for now not using config_get * change jedi build/source in platform * add code for genrating default dictionaries * fix the setup py paths * move task questions * initialize the task questions for components * init def dict atmos * use existing for platform * set the defaults * standardize the yamls * add geos directories * add new suites * name change in config * fixes to get hofx and 3dvar running * remove num iterations from RunJediHofx task * coding norms * add dictionary gneeration test * add test_suite action * action for test suite * add init to test path * copy task qustions to install dir * action job name * build jedi paths * make tasks complianct with new config and base delgations * Add questions to task_questions * Add questions to task_questions formatting * add question answers to model interface * Do not overwrite the dictionary in the source code * more checking * improve testing of dictionary creation * Use spack-stack 1.4 modules to maintain compatibility with JEDI * check for defer_to_ in the files * reduce prints in tests * py coding norms * unload crtm and update jedi_bundle * unused variables * rename suite * Bug fix and remove print * add operator test capability * rename some things * temp comment out * YAML generation working * add geos adas suite * Tasks for geosadas running * dates * unused variables * standardise task_questions * update default dictionaries * config changes * GMI_GPM full configuration for IODA v3 and x0048. (#187) * GMI_GPM full configuration for IODA v3 and x0048. * Code norm. * Code norm. * update data paths * all tasks running smoothly * rename workflows * coding norms * yaml norms * test suite * replace tier2 * tier2 name * tier 2 take 2 * remove trigger for tier1 * update target workflows * update target workflows branch * some renaming * action testing * action testing * bug fix in run_command print * bug fix in run_command print * action testing * action testing * flow bug * add alternative workflow that generates the geovals as well as obs * better requirements * add da run parameter * ufo_testing runs in cylc * update hofx to be x0048 * updating scatwind yaml (#194) * UFO Tests - Satwind (#193) * x48 satwind yaml * clean up and benchmark change * removing @s * yaml fix * add some missing instruments * add gps and avhrr specific cases * Feature/jjin3/ufo tests x0048 micorwave (#197) * Update configurations for microwave data in x0048. * Code norm. * Code norm. * Code norm. * Code norm. * Code norm. * YAML files for CrIS-FSR(n20,npp), AIRS & SSMIS (#200) Co-authored-by: Dan Holdaway <[email protected]> * clean up issues in ncdiag 2 ioda calling task * fix bug for single file outputs * updating YAML resource files for IASI (#202) * ozone working better * do not always create the forecast directory * Change the name of the filter assign observational error to "Perform Action". * revert to develop --------- Co-authored-by: danholdaway <[email protected]> Co-authored-by: Doruk Ardağ <[email protected]> Co-authored-by: dooruk <[email protected]> Co-authored-by: Jianjun Jin <[email protected]> Co-authored-by: Akira Sewnath <[email protected]> Co-authored-by: Wei Gu <[email protected]> Co-authored-by: gmao-jjin3 <[email protected]> * put suite files where they should be * can create experiment for all suites * override bug fixes * reorganize testing directory * generates the same 3dvar_cycle yaml as before * remove print * copy suite test yamls to install * clean up ocean yaml * work on 3dvar * same experiment yamls as develop * same experiment yamls as develop hofx * fix eva issue * Use existing build for var runs * allow users to override some slurm variables * allow for both test and use override * Remove some rogue system files * coding norms --------- Co-authored-by: danholdaway <[email protected]> Co-authored-by: Doruk Ardağ <[email protected]> Co-authored-by: dooruk <[email protected]> Co-authored-by: Jianjun Jin <[email protected]> Co-authored-by: Akira Sewnath <[email protected]> Co-authored-by: Wei Gu <[email protected]> Co-authored-by: gmao-jjin3 <[email protected]>
GEOS-ESM · Aug 14, 2023 · 22630f3 · 22630f3
1 parent e093a54
commit 22630f3
Show file tree

Hide file tree

Showing 87 changed files with 1,290 additions and 2,486 deletions.
diff --git a/.github/workflows/test_suite.yml → .github/workflows/code_tests.yml b/.github/workflows/test_suite.yml → .github/workflows/code_tests.yml
@@ -1,4 +1,4 @@
-name: Swell Test Suite
+name: Swell Code Testing
 
 on:
   push:
@@ -12,7 +12,7 @@ on:
 
 jobs:
   testsuite:
-    name: Run swell test suite
+    name: Run swell coding testing
     runs-on: ubuntu-latest
 
     steps:
@@ -49,6 +49,6 @@ jobs:
     - name: Put swell in the path
       run: echo "$HOME/.local/bin" >> $GITHUB_PATH
 
-    # Run the swell test suite
-    - name: Run swell test suite
-      run: swell_test_suite
+    # Run the swell code tests
+    - name: Run swell code tests
+      run: swell_tests_code
diff --git a/setup.py b/setup.py
@@ -39,6 +39,7 @@
                'suites/*/*',
                'suites/*/*/*',
                'tasks/task_questions.yaml',
+               'test/suite_tests/*.yaml',
                'configuration/*',
                'configuration/*/*',
                'configuration/*/*/*',
@@ -60,7 +61,7 @@
             'swell_util_task_question_dicts = swell.utilities.bin.task_question_dicts:tq_dicts',
             'swell_util_task_question_dicts_defaults = \
                 swell.utilities.bin.task_question_dicts_defaults:tq_dicts_defaults',
-            'swell_test_suite = swell.test.test_suite:main',
+            'swell_tests_code = swell.test.code_tests.code_tests:main',
         ],
     },
     )

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_aqua.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_aqua.yaml
@@ -78,7 +78,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-a.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-a.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-b.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-b.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-c.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_metop-c.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n15.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n15.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n18.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n18.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n19.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/amsua_n19.yaml
@@ -80,7 +80,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: 1-15

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/atms_n20.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/atms_n20.yaml
@@ -78,7 +78,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: atms_n20_channels

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/atms_npp.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/atms_npp.yaml
@@ -78,7 +78,7 @@ obs filters:
     action:
       name: reject
 # Assign obs error
-  - filter: Bounds Check
+  - filter: Perform Action
     filter variables:
     - name: brightnessTemperature
       channels: atms_npp_channels

diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/ompslpnc_npp.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/observations/ompslpnc_npp.yaml
diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/task_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/task_questions.yaml
@@ -18,7 +18,7 @@ background_error_model:
   - GSIbec
 
 background_experiment:
-  default_value: x0044
+  default_value: x0048
 
 background_frequency:
   default_value: PT1H
@@ -32,10 +32,6 @@ clean_patterns:
   - '*.txt'
   - logfile.*.out
 
-forecast_duration:
-  default_value:
-  - PT6H
-
 geos_background_restart_offset:
   default_value: TODO
 
@@ -46,7 +42,7 @@ geos_bkg_tar_filename_template:
   default_value: TODO
 
 geovals_experiment:
-  default_value: x0044_test_obs_filters_geovals
+  default_value: x0048-geovals
 
 geovals_provider:
   default_value: ncdiag
@@ -55,9 +51,9 @@ gradient_norm_reduction:
   default_value: 10e-5
 
 horizontal_resolution:
-  default_value: C361
+  default_value: '361'
   options:
-  - C361
+  - '361'
 
 jedi_bkg_filename_template:
   default_value: TODO
@@ -79,42 +75,79 @@ npy_proc:
   default_value: 2
 
 number_of_iterations:
-  default_value: 50
+  default_value:
+  - 50
 
 obs_experiment:
-  default_value: x0044_v3
+  default_value: x0048
 
 obs_provider:
   default_value: ncdiag
 
 observations:
   default_value:
   - aircraft
-  - amsua_n19
+  - gps
+  - satwind
+  - scatwind
+  - sfcship
+  - sfc
+  - sondes
+  - airs_aqua
+  - amsr2_gcom-w1
   - amsua_aqua
-  - amsua_metop-a
   - amsua_metop-b
   - amsua_metop-c
   - amsua_n15
   - amsua_n18
   - amsua_n19
+  - atms_n20
+  - atms_npp
+  - avhrr3_metop-b
+  - avhrr3_n18
+  - avhrr3_n19
+  - cris-fsr_n20
+  - cris-fsr_npp
   - gmi_gpm
-  - amsr2_gcom-w1
+  - iasi_metop-b
+  - iasi_metop-c
+  - mhs_metop-b
+  - mhs_metop-c
+  - mhs_n19
+  - ssmis_f17
   - mls55_aura
   - omi_aura
   - ompsnm_npp
   options:
   - aircraft
-  - amsua_n19
+  - gps
+  - satwind
+  - scatwind
+  - sfcship
+  - sfc
+  - sondes
+  - airs_aqua
+  - amsr2_gcom-w1
   - amsua_aqua
-  - amsua_metop-a
   - amsua_metop-b
   - amsua_metop-c
   - amsua_n15
   - amsua_n18
   - amsua_n19
+  - atms_n20
+  - atms_npp
+  - avhrr3_metop-b
+  - avhrr3_n18
+  - avhrr3_n19
+  - cris-fsr_n20
+  - cris-fsr_npp
   - gmi_gpm
-  - amsr2_gcom-w1
+  - iasi_metop-b
+  - iasi_metop-c
+  - mhs_metop-b
+  - mhs_metop-c
+  - mhs_n19
+  - ssmis_f17
   - mls55_aura
   - omi_aura
   - ompsnm_npp
@@ -123,12 +156,10 @@ path_to_geos_adas_background:
   default_value: /discover/nobackup/drholdaw/SwellTestData/geosadas/bkg/*bkg_clcv_rst*
 
 path_to_gsi_bc_coefficients:
-  default_value:
-  - /discover/nobackup/drholdaw/SwellTestData/geosadas/bc
+  default_value: /discover/nobackup/drholdaw/SwellTestData/ufo_testing/bc/2021121200
 
 path_to_gsi_nc_diags:
-  default_value:
-  - /discover/nobackup/drholdaw/SwellTestData/geosadas/obs
+  default_value: /discover/nobackup/drholdaw/SwellTestData/ufo_testing/ncdiag/2021121200
 
 produce_geovals:
   default_value: true
@@ -142,13 +173,15 @@ static_background_error_model:
   - GSIbec
 
 vertical_resolution:
-  default_value: 72
+  default_value: '72'
   options:
-  - 72
+  - '72'
 
 window_length:
   default_value: PT6H
 
 window_offset:
   default_value: PT3H
 
+window_type:
+  default_value: '4D'
diff --git a/src/swell/configuration/jedi/interfaces/geos_ocean/suite_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_ocean/suite_questions.yaml
@@ -1,3 +1,3 @@
 cycle_times:
   default_value: ['T00', 'T12']
-  options: ['T00', 'T06', 'T12', 'T18']
+  options: ['T00', 'T12']
diff --git a/src/swell/configuration/jedi/interfaces/geos_ocean/task_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_ocean/task_questions.yaml
@@ -29,10 +29,6 @@ clean_patterns:
   - '*MOM*'
   - logfile.*.out
 
-forecast_duration:
-  default_value:
-  - P1D
-
 gradient_norm_reduction:
   default_value: 1e-10
 
@@ -53,7 +49,8 @@ minimizer:
   - RPCG
 
 number_of_iterations:
-  default_value: 5
+  default_value:
+  - 5
 
 obs_experiment:
   default_value: s2s
@@ -80,13 +77,16 @@ total_processors:
   default_value: 24
 
 vertical_resolution:
-  default_value: 75
+  default_value: '75'
   options:
-  - 75
+  - '75'
 
 window_length:
   default_value: PT12H
 
 window_offset:
   default_value: PT6H
 
+window_type:
+  default_value: '3D'
+