Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new suite to run the UFO tests for the observation operator YAMLs #191

Merged
merged 137 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
137 commits
Select commit Hold shift + click to select a range
a3f2056
add suite for obs filter tests
danholdaway Mar 14, 2023
d542df0
Merge branch 'develop' into feature/ufo_tests
danholdaway Mar 22, 2023
f8bf7dd
add ncdiag tasks
danholdaway Mar 23, 2023
84f2957
converter working (no geovals)
danholdaway Mar 31, 2023
da8bb24
Merge branch 'develop' into feature/ufo_tests
danholdaway Apr 5, 2023
f7d02f4
executable runs
danholdaway Apr 11, 2023
e25bae1
sat bias correction conversion added
danholdaway Apr 12, 2023
adbe94a
add aircraft yaml
danholdaway Apr 12, 2023
e846b81
move to R2D2 for getting the observations
danholdaway Apr 13, 2023
3ac880a
convert to using GetObservations
danholdaway Apr 13, 2023
28c8577
append
danholdaway May 2, 2023
66791da
uncomment
danholdaway May 10, 2023
d77cf3e
Merge branch 'develop' into feature/ufo_tests
danholdaway May 10, 2023
399e5ac
clean up hidden aspects of the config
danholdaway May 15, 2023
78c171d
move the opening of jedi interface files
danholdaway May 15, 2023
b118ee6
add class to handle jedi file templating
danholdaway May 19, 2023
23d90e9
stage tasks working
danholdaway May 19, 2023
eda29db
unify copyright date
danholdaway May 22, 2023
a28793f
pep8
danholdaway May 22, 2023
35e574d
add utility for setting the list of tasks for each question
danholdaway May 23, 2023
dbb7625
add question prompts
danholdaway May 24, 2023
c54565c
add more questions and defaults
danholdaway May 24, 2023
c8ccdd5
move jedi exe base
danholdaway May 24, 2023
0abbd51
refactor run jedi exe base
danholdaway May 24, 2023
c6fe46f
remove fgat as not needed
danholdaway May 24, 2023
92c884b
move depends
danholdaway May 24, 2023
f80c852
write lists the way python does
danholdaway May 24, 2023
7a53252
make sure jedi_forecast model is used
danholdaway May 24, 2023
0f67072
more q clean up
danholdaway May 24, 2023
8aacda5
remove jedi exe base class from questions
danholdaway May 24, 2023
ff0c3f9
convert task names to CamelCase in the questions
danholdaway May 24, 2023
6711c2e
modify config class to create variables and methods to get them
danholdaway May 24, 2023
b9fc302
Update generate_b_climatology_by_linking.py
Dooruk May 24, 2023
0bde116
common config availability
danholdaway May 24, 2023
e66856f
eva obs fix
Dooruk May 24, 2023
b3d175b
use new variable functions for config instead of config_get
danholdaway May 24, 2023
d0dba9f
fix clean patterns
danholdaway May 24, 2023
a9fc049
update question yaml generation for now not using config_get
danholdaway May 24, 2023
2f0d602
change jedi build/source in platform
danholdaway May 24, 2023
ed47518
add code for genrating default dictionaries
danholdaway May 25, 2023
9412c67
fix the setup py paths
danholdaway May 25, 2023
ea33459
move task questions
danholdaway May 25, 2023
c0fa630
initialize the task questions for components
danholdaway May 25, 2023
a190ff3
init def dict atmos
danholdaway May 25, 2023
c368a52
use existing for platform
danholdaway May 25, 2023
d3785ec
set the defaults
danholdaway May 25, 2023
fd5aa83
standardize the yamls
danholdaway May 25, 2023
41949ad
add geos directories
danholdaway May 25, 2023
cdeabca
add new suites
danholdaway May 25, 2023
f83cbcd
name change in config
danholdaway May 25, 2023
0cf378a
fixes to get hofx and 3dvar running
danholdaway Jun 2, 2023
d8f1297
remove num iterations from RunJediHofx task
danholdaway Jun 2, 2023
1fd3ddb
coding norms
danholdaway Jun 2, 2023
af993de
add dictionary gneeration test
danholdaway Jun 2, 2023
2c65996
add test_suite action
danholdaway Jun 2, 2023
a184c84
Merge branch 'develop' into feature/clean_up_config
danholdaway Jun 2, 2023
b75c056
Merge branch 'feature/clean_up_config' into feature/questions_to_tasks
danholdaway Jun 2, 2023
08b0503
action for test suite
danholdaway Jun 2, 2023
c5b2218
add init to test path
danholdaway Jun 2, 2023
f313213
copy task qustions to install dir
danholdaway Jun 2, 2023
8983257
action job name
danholdaway Jun 2, 2023
35a527e
build jedi paths
danholdaway Jun 5, 2023
2276454
Merge branch 'feature/questions_to_tasks' into feature/ufo_tests
danholdaway Jun 5, 2023
0901967
make tasks complianct with new config and base delgations
danholdaway Jun 5, 2023
23c80db
Add questions to task_questions
danholdaway Jun 5, 2023
bf07cf9
Add questions to task_questions formatting
danholdaway Jun 5, 2023
f049afe
add question answers to model interface
danholdaway Jun 5, 2023
ac93f14
Do not overwrite the dictionary in the source code
danholdaway Jun 6, 2023
8420120
more checking
danholdaway Jun 7, 2023
a4db01c
improve testing of dictionary creation
danholdaway Jun 7, 2023
23a66cf
Use spack-stack 1.4 modules to maintain compatibility with JEDI
danholdaway Jun 7, 2023
e14ee13
check for defer_to_ in the files
danholdaway Jun 7, 2023
a483a09
reduce prints in tests
danholdaway Jun 7, 2023
d515153
py coding norms
danholdaway Jun 7, 2023
d7c026b
Merge branch 'develop' into feature/ss-1.4
danholdaway Jun 7, 2023
8453c27
unload crtm and update jedi_bundle
danholdaway Jun 9, 2023
4afd266
Merge branch 'develop' into feature/questions_to_tasks
danholdaway Jun 9, 2023
f67e0ad
Merge branch 'feature/questions_to_tasks' into feature/ufo_tests
danholdaway Jun 9, 2023
cccaebb
Merge branch 'feature/ss-1.4' into feature/ufo_tests
danholdaway Jun 9, 2023
abed471
Merge branch 'develop' into feature/ufo_tests
danholdaway Jun 14, 2023
69158fb
unused variables
danholdaway Jun 14, 2023
ddfc102
rename suite
danholdaway Jun 14, 2023
09abaf7
Bug fix and remove print
danholdaway Jun 15, 2023
d40fbb2
add operator test capability
danholdaway Jun 17, 2023
205689e
rename some things
danholdaway Jun 18, 2023
36a25b3
temp comment out
danholdaway Jun 20, 2023
418cf94
Merge branch 'develop' into feature/ufo_tests
danholdaway Jun 20, 2023
9d0a80a
YAML generation working
danholdaway Jun 21, 2023
4f8be99
add geos adas suite
danholdaway Jun 21, 2023
1a04385
Merge branch 'develop' into feature/ufo_tests_rt
danholdaway Jun 21, 2023
2ed822a
Tasks for geosadas running
danholdaway Jun 27, 2023
70b8fec
dates
danholdaway Jun 27, 2023
fc7cd3c
unused variables
danholdaway Jun 27, 2023
9abd3b7
standardise task_questions
danholdaway Jun 27, 2023
87d43fa
update default dictionaries
danholdaway Jun 27, 2023
e89fae8
config changes
danholdaway Jun 28, 2023
6471d38
GMI_GPM full configuration for IODA v3 and x0048. (#187)
gmao-jjin3 Jul 12, 2023
3f17cd7
update data paths
danholdaway Jul 12, 2023
c69941b
Merge branch 'develop' into feature/ufo_tests
danholdaway Jul 12, 2023
b05fe36
all tasks running smoothly
danholdaway Jul 14, 2023
6cd4b57
rename workflows
danholdaway Jul 14, 2023
61f124b
coding norms
danholdaway Jul 14, 2023
82ef570
yaml norms
danholdaway Jul 14, 2023
12cdf58
test suite
danholdaway Jul 17, 2023
815637e
replace tier2
danholdaway Jul 17, 2023
0c8d2f4
tier2 name
danholdaway Jul 17, 2023
71cb165
tier 2 take 2
danholdaway Jul 17, 2023
303b8c6
remove trigger for tier1
danholdaway Jul 17, 2023
6b6dc1d
Merge branch 'develop' into feature/ufo_tests
danholdaway Jul 17, 2023
6ddf8cf
update target workflows
danholdaway Jul 17, 2023
72e4a71
update target workflows branch
danholdaway Jul 17, 2023
de6aa44
some renaming
danholdaway Jul 17, 2023
736c868
action testing
danholdaway Jul 18, 2023
3bb21f9
action testing
danholdaway Jul 18, 2023
0c3e4ff
bug fix in run_command print
danholdaway Jul 19, 2023
dd13a66
bug fix in run_command print
danholdaway Jul 19, 2023
4a3ab6b
action testing
danholdaway Jul 19, 2023
7a1aefb
action testing
danholdaway Jul 19, 2023
60b1172
flow bug
danholdaway Jul 19, 2023
5005029
add alternative workflow that generates the geovals as well as obs
danholdaway Jul 21, 2023
e740eed
better requirements
danholdaway Jul 21, 2023
8913ab9
add da run parameter
danholdaway Jul 21, 2023
73be5cd
ufo_testing runs in cylc
danholdaway Jul 21, 2023
e54c1e1
update hofx to be x0048
danholdaway Jul 24, 2023
c2fcc59
updating scatwind yaml (#194)
asewnath Jul 26, 2023
be27fa5
UFO Tests - Satwind (#193)
asewnath Jul 26, 2023
6317356
add some missing instruments
danholdaway Jul 26, 2023
5560e0d
add gps and avhrr specific cases
danholdaway Jul 26, 2023
b043030
Feature/jjin3/ufo tests x0048 micorwave (#197)
gmao-jjin3 Jul 27, 2023
5d3b8ca
Merge branch 'develop' into feature/ufo_tests
danholdaway Jul 27, 2023
a5bd337
YAML files for CrIS-FSR(n20,npp), AIRS & SSMIS (#200)
gmao-wgu Jul 27, 2023
be042c6
clean up issues in ncdiag 2 ioda calling task
danholdaway Jul 27, 2023
d68390e
fix bug for single file outputs
danholdaway Jul 27, 2023
c2add36
updating YAML resource files for IASI (#202)
gmao-wgu Jul 28, 2023
f78cfe0
ozone working better
danholdaway Aug 8, 2023
c934e62
do not always create the forecast directory
danholdaway Aug 8, 2023
a5a041b
more updates towards trifecta
danholdaway Aug 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/tier1_application_discover.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ defaults:
shell: bash

jobs:
run-discover-nightly:
uses: GEOS-ESM/CI-workflows/.github/workflows/swell-discover-nightly.yml@main
run-tier1-apps-discover:
uses: GEOS-ESM/CI-workflows/.github/workflows/swell-tier1_application_discover.yml@feature/ufo_tests
4 changes: 2 additions & 2 deletions .github/workflows/tier2_application_discover.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ defaults:
shell: bash

jobs:
run-discover-nightly:
uses: GEOS-ESM/CI-workflows/.github/workflows/swell-discover-nightly.yml@main
run-tier2-apps-discover:
uses: GEOS-ESM/CI-workflows/.github/workflows/swell-tier2_application_discover.yml@feature/ufo_tests
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# System files
*.swp
*DS_Store*
._*

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

### Licence:

(C) Copyright 2021-2022 United States Government as represented by the Administrator of the National
(C) Copyright 2021- United States Government as represented by the Administrator of the National
Aeronautics and Space Administration. All Rights Reserved.

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
Expand Down
12 changes: 12 additions & 0 deletions requirements-standalone.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
click>=8.1.5
jinja2>=3.1.2
pyyaml>=6.0
pycodestyle>=2.10.0
pandas>=1.4.0
isodate>=0.5.4
f90nml>=1.4.3
questionary>=1.10.0
flake8>=6.0.0
netCDF4
xarray
matplotlib
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
click>=8.1.5
jinja2>=3.1.2
pyyaml>=6.0
pycodestyle>=2.8.0
pycodestyle==2.8.0
pandas>=1.4.0
isodate>=0.5.4
f90nml>=1.4.3
Expand Down
2 changes: 1 addition & 1 deletion src/swell/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@
repo_directory = os.path.dirname(__file__)

# Set the version for swell
__version__ = '1.5.0'
__version__ = '1.6.0'
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
filetype: cube sphere history
provider: geos
datapath: '{{cycle_dir}}'
filename: '{{experiment_id}}.analysis.%yyyy%mm%dd_%hh%MM%ssz.nc4'
first: PT0H
frequency: PT1H
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ provider: geos
datapath: ''
filenames: ['{{cycle_dir}}/bkg.%yyyy%mm%ddT%hh%MM%ssZ.nc4',
'{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/bkg/geos.crtmsrf.{{horizontal_resolution}}.nc4']
state variables: [u,v,ua,va,t,delp,q,qi,ql,qr,qs,o3ppmv,phis,
state variables: [u,v,ua,va,t,delp,ps,q,qi,ql,qr,qs,o3ppmv,phis,
qls,qcn,cfcn,frocean,frland,varflt,ustar,bstar,
zpbl,cm,ct,cq,kcbl,tsm,khl,khu,frlake,frseaice,vtype,
stype,vfrac,sheleg,ts,soilt,soilm,u10m,v10m]
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
covariance model: SABER
saber central block:
saber block name: gsi covariance
read:
gsi akbk: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/akbk{{vertical_resolution}}.nc4'
gsi error covariance file: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/gsibec_coefficients_c{{horizontal_resolution}}.nc4'
gsi berror namelist file: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/gsibec_configuration_c{{horizontal_resolution}}.nml'
processor layout x direction: {{gsibec_npx_proc}}
processor layout y direction: {{gsibec_npy_proc}}
debugging mode: false
saber outer blocks:
- saber block name: gsi interpolation to model grid
state variables to inverse: &bvars [eastward_wind,northward_wind,air_temperature,surface_pressure,
specific_humidity,cloud_liquid_ice,cloud_liquid_water,
mole_fraction_of_ozone_in_air,
skin_temperature,
fraction_of_ocean,fraction_of_lake,fraction_of_ice,
sfc_geopotential_height_times_grav]
gsi akbk: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/akbk{{vertical_resolution}}.nc4'
gsi error covariance file: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/gsibec_coefficients_c{{horizontal_resolution}}.nc4'
gsi berror namelist file: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/gsibec_configuration_c{{horizontal_resolution}}.nml'
processor layout x direction: {{gsibec_npx_proc}}
processor layout y direction: {{gsibec_npy_proc}}
debugging mode: false
linear variable change:
linear variable change name: Control2Analysis
input variables: *bvars
output variables: {{analysis_variables}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
fms initialization:
namelist filename: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/fmsmpp.nml'
field table filename: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/field_table_gmao'
akbk: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/akbk{{vertical_resolution}}.nc4'
layout: [{{npx_proc}},{{npy_proc}}]
npx: {{horizontal_resolution}}
npy: {{horizontal_resolution}}
npz: {{vertical_resolution}}
field metadata override: '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fieldmetadata/geos.yaml'
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,3 @@ provider: geos
datapath: ''
filenames: ['{{cycle_dir}}/bkg.%yyyy%mm%ddT%hh%MM%ssZ.nc4',
'{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/bkg/geos.crtmsrf.{{horizontal_resolution}}.nc4']
model variables: [u,v,ua,va,t,delp,q,qi,ql,qr,qs,o3ppmv,phis,qls,qcn,cfcn,frocean,frland,varflt,ustar,
bstar,zpbl,cm,ct,cq,kcbl,tsm,khl,khu,frlake,frseaice,vtype,stype,vfrac,sheleg,
ts,soilt,soilm,u10m,v10m]
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@
- ['{{swell_static_files}}/jedi/interfaces/geos_atmosphere/GEOS_CRTM_Surface/geos.crtmsrf.{{horizontal_resolution}}.nc4', '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/bkg/']
- ['{{experiment_root}}/{{experiment_id}}/jedi_bundle/source/fv3-jedi/test/Data/fieldmetadata/*', '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fieldmetadata/']
- ['{{experiment_root}}/{{experiment_id}}/jedi_bundle/source/fv3-jedi/test/Data/fv3files/*', '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/fv3files/']
- ['{{swell_static_files}}/jedi/interfaces/geos_atmosphere/gsibec/gsibec_coefficients_c{{horizontal_resolution}}.nc4', '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/']
- ['{{swell_static_files}}/jedi/interfaces/geos_atmosphere/gsibec/gsibec_configuration_c{{horizontal_resolution}}.nml', '{{experiment_root}}/{{experiment_id}}/stage/fv3-jedi/geos_atmosphere/gsibec/']
Loading