Single-species carbon simulations still not working for CO, CO2, and OCS in GCHP

[msulprizio]

Name and Institution (Required)

Name: Melissa Sulprizio
Institution: Harvard

Description of your issue or question

Even after the merge of #1992, the single species carbon simulations for CO, CO2, and OCS are still failing in GCHP. This was revealed by adding these simulations to integration tests in #2236. The issue is in the run directory creation for these simulations. Wildcards do not work in HISTORY.rc for GCHP so each species must be listed under the diagnostic collection's fields. For carbon, this looks like:

#==============================================================================
  SpeciesConc.template:       '%y4%m2%d2_%h2%n2z.nc4',
  SpeciesConc.format:         'CFIO',
  SpeciesConc.timestampStart: .true.
  SpeciesConc.monthly:        1
  SpeciesConc.frequency:      010000
  SpeciesConc.duration:       240000
  SpeciesConc.mode:           'time-averaged'
  SpeciesConc.fields:         'SpeciesConcVV_CH4',  'GCHPchem',
                              'SpeciesConcVV_CO ',  'GCHPchem',
                              'SpeciesConcVV_CO2',  'GCHPchem',
                              'SpeciesConcVV_OCS',  'GCHPchem',

The code in singleCarbonSpecies.sh updates HISTORY.rc to remove the carbon species that aren't needed.

geos-chem/run/shared/singleCarbonSpecies.sh

Lines 218 to 225 in 64796d8

	# File to be modified
	file="${2}/HISTORY.rc"
	# For GCHP: remove entries for species to be excluded
	for spc in ${1}; do
	sed -i "/\_${spc}/d" "${file}"
	sed -i "/Emis${spc}/d" "${file}"
	done

However, this results in the removal of the line

  SpeciesConc.fields:         'SpeciesConcVV_CH4',  'GCHPchem',

for CO, CO2, and OCS simulations and therefore no fields defined for the SpeciesConc collection. For example, the resulting HISTORY.rc for CO looks like:

#==============================================================================
  SpeciesConc.template:       '%y4%m2%d2_%h2%n2z.nc4',
  SpeciesConc.format:         'CFIO',
  SpeciesConc.timestampStart: .true.
  SpeciesConc.monthly:        1
  SpeciesConc.frequency:      010000
  SpeciesConc.duration:       240000
  SpeciesConc.mode:           'time-averaged'
                              'SpeciesConcVV_CO ',  'GCHPchem',
                              ##'SpeciesConcMND_CO ', 'GCHPchem',
::

In addition, for CO2 simulations instances of _CO2 and EmisCO2 are also removed from the configuration files because they include "CO". The resulting SpeciesConc collection in HISTORY.rc for CO2 looks like:

#==============================================================================
  SpeciesConc.template:       '%y4%m2%d2_%h2%n2z.nc4',
  SpeciesConc.format:         'CFIO',
  SpeciesConc.timestampStart: .true.
  SpeciesConc.monthly:        1
  SpeciesConc.frequency:      010000
  SpeciesConc.duration:       240000
  SpeciesConc.mode:           'time-averaged'
::

[yantosca]

@msulprizio: Thanks for letting me know about this. I am testing a fix:

function updateHistory() {

    #========================================================================
    # Removes entries in HISTORY.rc for unused species
    #
    # Arguments:
    # ${1} : List of species to exclude
    # ${2} : Path to the run directory
    #========================================================================

    # File to be modified
    file="${2}/HISTORY.rc"

    # For GCHP: remove entries for species to be excluded
    for spc in ${1}; do
	sed -i "/\_${spc}'/d"   "${file}"
        sed -i "/\_${spc} '/d"  "${file}"
        sed -i "/Emis${spc}_/d"  "${file}"
    done

    # Also disable emissions for OCS-only simulations
    # (as we currently do not have any)
    isItemInList "OCS" "${1}"
    if [[ $? == 1 ]]; then
	sed -i -e "s/'Emissions/#'Emissions/" "${file}"
    fi
}

[yantosca]

Actually the code above didn't work. This is the proper sed command:

	sed -i "/\_${spc}[ \t]*'/d"  "${file}"

the [ \t]* removes an zero or more spaces and/or tabs.

[yantosca]

@msulprizio: @lizziel: I've made a few more fixes to the single-tracer simulation (see commits 77b89f5, a2db197, 371c8e3, 39a103c, and 58aeb1a) but the run seems to error out during the init phase of the History component. The error I get is similar to what @amy1916 is seeing in #2192.

Relevant config & log files are here: carbon-sim.zip

The error that I get (traceback condensed for clarity) is:

Writing output to gchp.20190101_0000z.log

Currently Loaded Modules:
  1) gmp/6.2.1-fasrc01       7) szip/2.1.1-fasrc01
  2) mpfr/4.1.0-fasrc01      8) hdf5/1.10.7-fasrc01
  3) mpc/1.2.1-fasrc01       9) netcdf-c/4.8.0-fasrc01
  4) gcc/10.2.0-fasrc01     10) netcdf-fortran/4.5.3-fasrc01
  5) openmpi/4.1.0-fasrc01  11) flex/2.6.4-fasrc01
  6) zlib/1.2.11-fasrc01    12) cmake/3.25.2-fasrc01

pe=00000 FAIL at line=03325    MAPL_HistoryGridComp.F90                 <status=41>
pe=00000 FAIL at line=00958    MAPL_HistoryGridComp.F90                 <status=41>
pe=00000 FAIL at line=01807    MAPL_Generic.F90                         <status=41>
pe=00000 FAIL at line=00713    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=00658    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=00959    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=00311    MAPL_Cap.F90                             <status=41>
pe=00000 FAIL at line=00258    MAPL_Cap.F90                             <status=41>
pe=00000 FAIL at line=00192    MAPL_Cap.F90                             <status=41>
pe=00000 FAIL at line=00169    MAPL_Cap.F90                             <status=41>
pe=00000 FAIL at line=00029    GCHPctm.F90                              <status=41>

...
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 27065758.0 ON holy8a24105 CANCELLED AT 2024-04-05T17:41:41 ***
srun: error: holy8a24105: task 0: Exited with exit code 112
srun: error: holy8a24105: task 1: Exited with exit code 160
srun: error: holy8a24105: tasks 2,4-5: Exited with exit code 176

real	0m2.176s
user	0m0.026s
sys	0m0.021s

As in #2192, this is crashing in this section of MAPL_HistoryGridComp.F90:

https://github.com/geoschem/MAPL/blob/d61f89a7ab2a2691bc2b0dd1a42b2871c6420740/gridcomps/History/MAPL_HistoryGridComp.F90#L3302-L3325

NOTE: I had added a print statement in my MAPL code folder so line 3325 is actually line 3324 as shown above. There is some issue with parsing the HISTORY.rc file. I will dig a bit further and add more printout.

[yantosca]

On further review, the issue is caused by the .fields tag being deleted when removing HISTORY.rc entries for CH4. by the singleCarbonSpecies.sh script. This left the e.g. Emissions collection definition looking like this:

  Emissions.template:         '%y4%m2%d2_%h2%n2z.nc4',
  Emissions.format:           'CFIO',
  Emissions.timestampStart:   .true.
  Emissions.monthly:          0
  Emissions.frequency:        010000
  Emissions.duration:         010000
  Emissions.mode:             'time-averaged'
                              'EmisCO2_Total        ', 'GCHPchem',
                              'EmisCO2_FossilFuel   ', 'GCHPchem',
                              'EmisCO2_Ocean        ', 'GCHPchem',
                              'EmisCO2_BalBiosph    ', 'GCHPchem',
                              'EmisCO2_NetTerrExch  ', 'GCHPchem',
                              'EmisCO2_Ship         ', 'GCHPchem',
                              'EmisCO2_Aviation     ', 'GCHPchem',
                              'EmisCO2_CO2SurfCorr  ', 'GCHPchem',
                              'EmisCO2_BiomassBurn  ', 'GCHPchem',

We will have to put in some more logic to work around this.

@lizziel: Do you know if the line containing the .fields tag has to be populated? Could it be a comment?

[yantosca]

The best solution might be to have different HISTORY.rc files for the single-species GCHP carbon simulations, and to have the createRunDir.sh script place the relevant file in the run directory.

[msulprizio]

This has been resolved by #2236.