Cycling FGAT for geos_marine and towards erasing geos_ocean (#419)

* executable and a brand new task

* new task and dynamic keys as a new option

* make suite to run accesible

* soft run option for generate B climatology

* fgat cycle suite and template yamls

* eva change

* codestyle fixes

* allow setting env variable with code tests

* add fgat executable and reorganize linking according to models

* code test fixes

* create marine_models key as a SOCA related tasks control switch

* use marine_models in templates to control config behavoir

* fix default typehinting

* update config to use generic saber

* adapt to using cwd, include **kwargs in shell utils

* update link_geos for cice6 history handling

* fgat with geos_marine

* fix typehinting

* include analysis time as a template variable

* eva related changes for increment outputs

* preaching docs, gotta add docs

docs/configs/model_configurations/cice6.md

@@ -0,0 +1,31 @@
+# CICE6 Settings for GEOS/SOCA Setup (CICE_6.4.1/GEOSv11.6)
+
+SOCA currently only uses category aggregated CICE variables, namely:
+
+- `cicen`: Ice are (sea-ice concentration)
+- `aicen`: Ice thickness
+- `hsnon`: Snow thickness
+
+
+## History Outputs
+
+>In SWELL, history outputs are utilized to obtain the aggregated variables in desired output frequencies. See more details about [history outputs](history_outputs.md).
+
+The following setup is for using 3 hourly states. In `ice_in`, `histfreq` should have `h` and `histfreq_n`
+should be 3 for 3 hourly dumps.
+
+```nml
+histfreq       = 'h','x','x','x','x'
+histfreq_n     =  3 , 1 , 1 , 1 , 1
+```
+
+Then, in the ` &icefields_nml` section, activate following variables:
+
+```nml
+f_aice         = 'h'
+f_hi           = 'h'
+f_hs           = 'h'
+```
+
+This will produce outputs named such as `iceh_03h.2021-07-02-43200.nc`, with `03h` prefix depending on the
+`histfreq` setup.

docs/configs/model_configurations/history_outputs.md

@@ -0,0 +1,47 @@
+History outputs, as opposed to model restarts containing full diagnostic fields, provides a subset of variables in preset frequencies. They are typically used for providing more frequent outputs of reduced dimension variables for plotting or analysis (i.e., vertically averaged or surface-only, such as SST).
+
+GEOSgcm provides a mechanism for dumping restarts (aka `_checkpoint`) more frequently via `Restart Record Parameters` section in `AGCM.rc`. However, this causes model to stop and create these large files with full model fields which is undesirable for long simulation times.
+
+In SWELL context, history outputs are particularly useful for 4D window setups when multiple states and/or backgrounds are required as an input for JEDI. For instance, for a 3DFGAT setup with a 6h window and 3h background frequency, one background (at the beginning of the window) and two state inputs (one at the 3h mark one at the end) are required. This can be handled by using `diag_table` for [MOM6](mom6.md) or adjusting `ice_in` for [CICE6](cice6.md).
+
+For example, with MOM6/CICE6, employing a 6h DA window and 3h background frequency via 3DFGAT, ocean and sea-ice backgrounds are defined by:
+
+```yaml
+background:
+  basename: ./
+  date: '2021-07-02T03:00:00Z'
+  ice_filename: cice.res.20210702T030000Z.nc
+  ocn_filename: MOM6.res.20210702T030000Z.nc
+  read_from_file: 1
+  state variables:
+  - cicen
+  - hicen
+  - hsnon
+  - socn
+  - tocn
+  - ssh
+  - hocn
+  - mld
+  - layer_depth
+```
+
+and ocean and sea-ice states:
+
+```yaml
+model:
+  name: PseudoModel
+  states:
+  - basename: ./
+    date: '2021-07-02T06:00:00Z'
+    ice_filename: ice.fc.2021-07-02T03:00:00Z.PT3H.nc
+    ocn_filename: ocn.fc.2021-07-02T03:00:00Z.PT3H.nc
+    read_from_file: 1
+  - basename: ./
+    date: '2021-07-02T09:00:00Z'
+    ice_filename: ice.fc.2021-07-02T03:00:00Z.PT6H.nc
+    ocn_filename: ocn.fc.2021-07-02T03:00:00Z.PT6H.nc
+    read_from_file: 1
+  tstep: PT3H
+```
+
+More details can be found in [JEDI/SOCA documentation](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/jedi-components/soca/index.html).

docs/configs/model_configurations/mom6.md

@@ -0,0 +1,8 @@
+# MOM6 Settings for GEOS/SOCA Setup (GEOSv11.6)
+
+
+## History Outputs
+
+>In SWELL, history outputs are utilized to obtain the aggregated variables in desired output frequencies. See more details about [history outputs](history_outputs.md).
+
+TODO: `diag_table`

src/swell/configuration/jedi/interfaces/geos_marine/geos_marine.yaml

@@ -1,6 +1,7 @@
 jedi_interface: soca
 total_processors: {{total_processors}}
 executables:
+  fgat: soca_var.x
   hofx3D: soca_hofx3d.x
   hofx4D: soca_hofx.x
   variational3D: soca_var.x

src/swell/configuration/jedi/interfaces/geos_marine/model/background.yaml

@@ -2,5 +2,7 @@ date: '{{local_background_time_iso}}'
 read_from_file: 1
 basename: './'
 ocn_filename: 'MOM6.res.{{local_background_time}}.nc'
+{% if 'cice6' in marine_models %}
 ice_filename: 'cice.res.{{local_background_time}}.nc'
+{% endif %}
 state variables: [cicen, hicen, hsnon, socn, tocn, ssh, hocn, mld, layer_depth]

src/swell/configuration/jedi/interfaces/geos_marine/model/background_diffusion_vt.yaml

@@ -0,0 +1,8 @@
+date: '{{local_background_time_iso}}'
+read_from_file: 1
+basename: './'
+ocn_filename: 'MOM6.res.{{local_background_time}}.nc'
+{% if 'cice6' in marine_models %}
+ice_filename: 'cice.res.{{local_background_time}}.nc'
+{% endif %}
+state variables: [tocn]

src/swell/configuration/jedi/interfaces/geos_marine/model/background_error.yaml

@@ -1,38 +1,31 @@
 covariance model: SABER
 saber central block:
-  saber block name: EXPLICIT_DIFFUSION
+  saber block name: diffusion
   active variables: [cicen, hicen, hsnon, tocn, socn, ssh]
-  geometry:
-    mom6_input_nml: 'soca/input.nml'
-    fields metadata: 'soca/fields_metadata.yaml'
-    geom_grid_file: 'INPUT/soca_gridspec.nc'
-  group mapping:
-  - name: ocn_3d
-    variables: [tocn, socn, cicen, hicen, hsnon]
-  - name: ocn_2d
-    variables: [ssh]
   read:
     groups:
-    - name: ocn_3d
+    - variables: [tocn, socn]
       horizontal:
-        filename: 'background_error_model/hz_rossby.nc'
+        filepath: 'background_error_model/hz_rossby'
       vertical:
-        filename: 'background_error_model/vt.{{local_background_time}}.nc'
-    - name: ocn_2d
+        levels: {{vertical_resolution}}
+        filepath: 'background_error_model/vt.{{local_background_time}}'
+    - variables: [ssh, cicen, hicen, hsnon]
       horizontal:
-        filename: 'background_error_model/hz_rossby_1p5.nc'
+        filepath: 'background_error_model/hz_rossby_1p5'
 
 date: '{{local_background_time_iso}}'
 
-linear variable change:
-  input variables: {{analysis_variables}}
-  output variables: {{analysis_variables}}
-  linear variable changes:
-  - linear variable change name: BkgErrFILT
+saber outer blocks:
+  - saber block name: SOCABkgErrFilt
     ocean_depth_min: 100 # [m]
     rescale_bkgerr: 1.0
     efold_z: 2500.0       # [m]
 
+linear variable change:
+  input variables: {{analysis_variables}}
+  output variables: {{analysis_variables}}
+  linear variable changes:
   - linear variable change name: BkgErrGODAS
     sst_bgerr_file: '{{cycle_dir}}/soca/godas_sst_bgerr.nc'
     # output:
@@ -50,7 +43,6 @@ linear variable change:
     cicen_max: 0.5
     hicen_min: 10.0
     hicen_max: 100.0
-
   - linear variable change name: BalanceSOCA
     ksshts:
       nlayers: 2

src/swell/configuration/jedi/interfaces/geos_marine/model/background_error_diffusion_vt.yaml

@@ -1,21 +1,19 @@
   covariance model: SABER
   saber central block:
-    saber block name: EXPLICIT_DIFFUSION
-    geometry:
-      mom6_input_nml: 'soca/input.nml'
-      fields metadata: 'soca/fields_metadata.yaml'
-      geom_grid_file: 'INPUT/soca_gridspec.nc'
+    saber block name: diffusion
     calibration:
       normalization:
         # NOTE, not actually used here, since the normalization spec is only used for hz
         method: randomization
         iterations: 1000
 
       groups:
-      - name: vt
+      - name: diffusion_vt
         vertical:
-          from file:
-            filename: '{{cycle_dir}}/calculated_scales.nc'
-            variable name: vt
+          model file:
+            date: '{{local_background_time_iso}}'
+            basename: './'
+            ocn_filename: 'calculated_scales.nc'
+          model variable: tocn
         write:
-          filename: 'background_error_model/vt.{{local_background_time}}.nc'
+          filepath: 'background_error_model/vt.{{local_background_time}}'

src/swell/configuration/jedi/interfaces/geos_marine/model/pseudo-model.yaml

@@ -0,0 +1,3 @@
+name: PseudoModel
+tstep: {{background_frequency}}
+states: {{states}}

src/swell/configuration/jedi/interfaces/geos_marine/model/states.yaml

@@ -1,9 +1,11 @@
 - input:
     read_from_file: 1
     basename: ./
-    ocn_filename: ocn.{{experiment_id}}.an.{{local_background_time_iso}}.nc
-    ice_filename: ice.{{experiment_id}}.an.{{local_background_time_iso}}.nc
-    date: '{{local_background_time_iso}}'
+    ocn_filename: ocn.{{experiment_id}}.an.{{analysis_time_iso}}.nc
+    {% if 'cice6' in marine_models %}
+    ice_filename: ice.{{experiment_id}}.an.{{analysis_time_iso}}.nc
+    {% endif %}
+    date: '{{analysis_time_iso}}'
     state variables: [tocn, socn, hocn, cicen, hicen, hsnon]
   output:
     datadir: ./

src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml

@@ -1,3 +1,12 @@
 cycle_times:
   default_value: ['T00', 'T12']
   options: ['T00', 'T12']
+
+ensemble_hofx_strategy:
+  default_value: 'serial'
+
+ensemble_hofx_packets:
+  default_value: 1
+
+skip_ensemble_hofx:
+  default_value: true

src/swell/configuration/jedi/interfaces/geos_marine/task_questions.yaml

@@ -24,7 +24,7 @@ background_experiment:
   default_value: s2s
 
 background_frequency:
-  default_value: PT1H
+  default_value: PT3H
 
 background_time_offset:
   default_value: PT9H
@@ -63,6 +63,16 @@ jedi_forecast_model:
   options:
   - NA
 
+marine_models:
+  default_value:
+  - mom6
+  - cice6
+  options:
+  - mom6
+  - cice6
+  - bgc
+  - ww3
+
 minimizer:
   default_value: RPCG
   options:

src/swell/configuration/jedi/interfaces/geos_ocean/geos_ocean.yaml

@@ -1,6 +1,7 @@
 jedi_interface: soca
 total_processors: {{total_processors}}
 executables:
+  fgat: soca_var.x
   hofx3D: soca_hofx3d.x
   hofx4D: soca_hofx.x
   variational3D: soca_var.x

src/swell/configuration/jedi/interfaces/geos_ocean/model/background_error.yaml

@@ -42,10 +42,5 @@ linear variable change:
     hicen_min: 10.0
     hicen_max: 100.0
   - linear variable change name: BalanceSOCA
-    # kst:
-    #   dsdtmax: 0.1
-    #   dsdzmin: 3.0e-6
-    #   dtdzmin: 1.0e-6
-    #   nlayers: 2
     ksshts:
       nlayers: 2

src/swell/configuration/jedi/interfaces/geos_ocean/model/pseudo-model.yaml

@@ -0,0 +1,3 @@
+name: PseudoModel
+tstep: {{background_frequency}}
+states: {{states}}

src/swell/configuration/jedi/oops/fgat.yaml

@@ -0,0 +1,37 @@
+cost function:
+  cost type: 3D-FGAT
+  jb evaluation: false
+  time window:
+    begin: '{{window_begin_iso}}'
+    end: '{{window_end_iso}}'
+    bound to include: begin
+  geometry:
+    TASKFILLgeometry
+  analysis variables: {{analysis_variables}}
+  model:
+    TASKFILLpseudo-model
+  background:
+    TASKFILLbackground
+  background error:
+    TASKFILLbackground_error
+  observations:
+    observers:
+      SPECIALobservations
+variational:
+  minimizer:
+    algorithm: {{minimizer}}
+  iterations:
+  - geometry: TASKFILLgeometry_inner
+    gradient norm reduction: '{{gradient_norm_reduction}}'
+    ninner: '{{number_of_iterations}}'
+    diagnostics:
+      departures: ombg
+    online diagnostics:
+      TASKFILLvarincrement1
+final:
+  diagnostics:
+    departures: oman
+  prints:
+    frequency: PT3H
+output:
+  TASKFILLanalysis

src/swell/deployment/create_experiment.py

@@ -475,6 +475,15 @@ def prepare_cylc_suite_jinja2(
             logger.abort('The suite file required cycling_varbc but ' +
                          'geos_atmosphere is not in the model components.')
 
+    # Marine model toggles (only for geos_marine)
+    if 'marine_models' in suite_file:
+        if 'geos_marine' in model_components:
+            render_dictionary['marine_models'] = \
+                experiment_dict['models']['geos_marine']['marine_models']
+        else:
+            logger.abort('The suite file required marine_models but ' +
+                         'geos_marine is not in the model components.')
+
     render_dictionary['scheduling'] = prepare_scheduling_dict(logger, experiment_dict, platform)
 
     # Default execution time limit for everthing is PT1H