Fix incorrect time slice read as boundary conditions in GC-Classic nested model

[jimmielin]

Name and Institution (Required)

Name: Haipeng Lin
Institution: Harvard Univ.

Reported by @nicholasbalasus (Harvard Univ.)

Confirm you have reviewed the following documentation

• Contributing guidelines

Describe the update

Boundary conditions used in the GEOS-Chem "Classic" nested model are being read from the wrong time slices in the source netCDF file.

The error is a forward 3-hour offset (e.g., 03:00 reads 06:00 data instead).

This is due to an incorrect time slice offset used in GET_BC_TIME in time_mod.F90. This update fixes this issue.

The update also fixes minor metadata issues (State_Chm%BoundaryCond is in kg/kg dry) and adds new log output to illustrate and identify the issue.

Bug description

Get_Boundary_Conditions in HCO_Utilities_GC_Mod.F90 is responsible for reading in boundary conditions in the GEOS-Chem "Classic" nested model. It uses time slices provided by GET_BC_TIME() in time_mod.F90, namely HHMMSS, to get the correct hour for boundary conditions:

   ! Find the proper time-slice to read from disk
   t_index = ( HHMMSS / 030000 ) + 1

...
      ! Get variable from HEMCO and store in local array
      CALL HCO_GC_GetPtr( Input_Opt, State_Grid, TRIM(v_name), Ptr3D, RC, &
                       TIDX=t_index, FOUND=FOUND )

The time indices are implied to be 3-hourly spaced throughout a day, with 1..8 corresponding to 00z..21z.

Because the boundary conditions are applied instantaneously, the time slices should align with the current model time:

...
            ! West BC
            DO I = 1, State_Grid%WestBuffer
               Spc(N)%Conc(I,J,L) = State_Chm%BoundaryCond(I,J,L,N)
            ENDDO
...

However, this is not the current behavior. The timestamp at the end of GET_BOUNDARY_CONDITIONS is not the actual timestamp used for read. To illustrate this, edit the log output line

      WRITE( 6, 140 ) STAMP
140   FORMAT( 'GET_BOUNDARY_CONDITIONS: Done applying BCs at ', a )

to this, which uses the actual HHMMSS read in the subroutine:

WRITE( 6, * ) 'GET_BOUNDARY_CONDITIONS: Done applying BCs at ', STAMP, ' using ', HHMMSS, t_index

The logs show that the incorrect offset is used except in the first hour:

Min and Max of each species in BC file [mol/mol]:
Species   1,      CH4: Min = 3.027744242E-07  Max = 2.310302079E-06
 GET_BOUNDARY_CONDITIONS: Done applying BCs at 2019/01/01 00:00using           0
           1
...
********************************************
* B e g i n   T i m e   S t e p p i n g !! *
********************************************

---> DATE: 2019/01/01  UTC: 00:00  X-HRS:      0.000000

...

---> DATE: 2019/01/01  UTC: 03:00  X-HRS:      3.000000
...
Min and Max of each species in BC file [mol/mol]:
 GET_BOUNDARY_CONDITIONS: Done applying BCs at 2019/01/01 03:00using       60000
           3
...
---> DATE: 2019/01/01  UTC: 06:00  X-HRS:      6.000000
...
 GET_BOUNDARY_CONDITIONS: Done applying BCs at 2019/01/01 06:00using       90000
           4

The expected choice for BCs is to continuously use slices 1..8, but slice 2 is skipped here. Then, all BCs are offset by 3-hours: at 03:00, HHMMSS = 060000, thus slice 3 and not slice 2 is read.

The cause of this is due to GET_BC_TIME using 3-hour ahead time slices:

       DATE = GET_TIME_AHEAD( 10800 )

instead of the current time.

@nicholasbalasus has helpfully plotted the output of CHEM_BOUNDARYCOND_CH4 and the underlying boundary condition files before the fix.

The three columns correspond to CHEM_BOUNDARYCOND_CH4, current SpeciesConc_CH4, and underlying BC files:

Note that State_Chm%BoundaryCond here at 07z does not match the 06z boundary conditions. Instead, it matches the 09z data. As State_Chm%BoundaryCond is only updated every 3-hours, it can be inferred that the boundary conditions at 06z to 09z are wrong and are being read ahead.

The 'smearing' of the boundary condition data in the species concentration array in the buffer zone (shown in middle panels) is a separate issue being diagnosed and a fix will be PRd as soon as possible.

Expected changes

Effects on full-chemistry simulations to be evaluated but may be significant especially over sunrise/sunset boundary.

Effects on methane inversions may affect posterior runs but because the Jacobian is computed on differences, the error introduced by a 3-hour offset in BCs may be minimal.

With this fix, the correct BC time slices are now read and reflected in the logs:

---> DATE: 2019/01/01  UTC: 03:00  X-HRS:      3.000000
...
 GET_BOUNDARY_CONDITIONS: Done applying BCs at 2019/01/01 03:00using       30000
           2

Other minor fixes are to State_Chm metadata. According to code

! Read species concentrations from NetCDF [mol/mol] and
! store in State_Chm%BoundaryCond in [kg/kg dry]

this was incorrectly noted as v/v.

Also to hide the Min and Max of each species in BC file [mol/mol]: header when not necessary (not verbose / FIRST).

Reference(s)

N/A

Related Github Issue(s)

Possibly #1889 but might not be the root cause.

Please link to the corresponding Github issue here. If fixing a bug, there should be an issue describing it with steps to reproduce.

[nicholasbalasus]

Thanks, @jimmielin! I can't test this for 14.2.1 (see #1888), but I will test it for 14.0.2 (used for the plots included in this PR).

[yantosca]

Thanks @jimmielin. We can get this into 14.2.1 (which may end up getting merged into 14.2.0).

[yantosca]

This looks good to merge. We also need a CHANGELOG.md update but I can add that.

[yantosca]

Integration tests are now running.

[jimmielin]

Thanks @yantosca! Sorry for missing the CHANGELOG requirement, I'll remember this for the future pull requests.

[yantosca]

After merging atop PR #1884 and PR geoschem/HEMCO#229, all GEOS-Chem integration tests passed (except for TOMAS, which is a known issue):

==============================================================================
GEOS-Chem Classic: Execution Test Results

GCClassic #8c6548e GEOS-Chem submod update: Merge PR #1884 (Updates for GEOS)
GEOS-Chem #b7eb4ff5d Merge PR #1891 (Fix incorrect time slice for nested-grid BC's)
HEMCO     #477c7e8 Merge PR #229 (Fix incorrect longitude definitions for HEMCO standalone)

Using 24 OpenMP threads
Number of execution tests: 26

Submitted as SLURM job: 64710205
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gc_05x0625_NA_47L_merra2_CH4........................Execute Simulation....PASS
gc_05x0625_NA_47L_merra2_fullchem...................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem..........................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem_TOMAS15..................Execute Simulation....FAIL
gc_4x5_47L_merra2_fullchem_TOMAS40..................Execute Simulation....FAIL
gc_4x5_merra2_aerosol...............................Execute Simulation....PASS
gc_4x5_merra2_carbon................................Execute Simulation....PASS
gc_4x5_merra2_CH4...................................Execute Simulation....PASS
gc_4x5_merra2_CO2...................................Execute Simulation....PASS
gc_4x5_merra2_fullchem..............................Execute Simulation....PASS
gc_4x5_merra2_fullchem_aciduptake...................Execute Simulation....PASS
gc_4x5_merra2_fullchem_APM..........................Execute Simulation....PASS
gc_4x5_merra2_fullchem_benchmark....................Execute Simulation....PASS
gc_4x5_merra2_fullchem_complexSOA...................Execute Simulation....PASS
gc_4x5_merra2_fullchem_complexSOA_SVPOA.............Execute Simulation....PASS
gc_4x5_merra2_fullchem_LuoWd........................Execute Simulation....PASS
gc_4x5_merra2_fullchem_marinePOA....................Execute Simulation....PASS
gc_4x5_merra2_fullchem_RRTMG........................Execute Simulation....PASS
gc_4x5_merra2_Hg....................................Execute Simulation....PASS
gc_4x5_merra2_metals................................Execute Simulation....PASS
gc_4x5_merra2_POPs_BaP..............................Execute Simulation....PASS
gc_4x5_merra2_tagCH4................................Execute Simulation....PASS
gc_4x5_merra2_tagCO.................................Execute Simulation....PASS
gc_4x5_merra2_tagO3.................................Execute Simulation....PASS
gc_4x5_merra2_TransportTracers......................Execute Simulation....PASS
gc_4x5_merra2_TransportTracers_LuoWd................Execute Simulation....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 24
Execution tests failed: 2
Execution tests not yet completed: 0

Also, all integration tests were zero-diff w/r/t the PR #1884 and PR geoschem/HEMCO#229, with the exception of APM, RRTMG diagnostics. Also TOMAS integration tests both failed.

Checking gc_05x0625_NA_47L_merra2_CH4
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_05x0625_NA_47L_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem_TOMAS15
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem_TOMAS40
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_aerosol
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_carbon
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_CH4
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_CO2
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_aciduptake
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_APM
   -> 2 differences found in OutputDir
      * GCC_14.2.1_r14/rundirs/gc_4x5_merra2_fullchem_APM/OutputDir/GEOSChem.Metrics.20190701_0000z.nc4 
        GCC_14.2.1_r15/rundirs/gc_4x5_merra2_fullchem_APM/OutputDir/GEOSChem.Metrics.20190701_0000z.nc4 
      * GCC_14.2.1_r14/rundirs/gc_4x5_merra2_fullchem_APM/OutputDir/GEOSChem.SpeciesConc.20190701_0000z.nc4 
        GCC_14.2.1_r15/rundirs/gc_4x5_merra2_fullchem_APM/OutputDir/GEOSChem.SpeciesConc.20190701_0000z.nc4 
   -> 1 difference found in Restarts
      * GCC_14.2.1_r14/rundirs/gc_4x5_merra2_fullchem_APM/Restarts/GEOSChem.Restart.20190701_0100z.nc4 
        GCC_14.2.1_r15/rundirs/gc_4x5_merra2_fullchem_APM/Restarts/GEOSChem.Restart.20190701_0100z.nc4 

Checking gc_4x5_merra2_fullchem_benchmark
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_complexSOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_complexSOA_SVPOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_LuoWd
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_marinePOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_RRTMG
   -> 1 difference found in OutputDir
      * GCC_14.2.1_r14/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
        GCC_14.2.1_r15/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
   -> No differences in Restarts

Checking gc_4x5_merra2_Hg
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_metals
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_POPs_BaP
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagCH4
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagCO
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagO3
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_TransportTracers
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_TransportTracers_LuoWd
   -> No differences in OutputDir
   -> No differences in Restarts

[yantosca]

After merging atop PR #1884 and HEMCO PR geoschem/HEMCO#229, all GEOS-Chem integration tests passed (except for TOMAS, which is a known issue):

==============================================================================
GCHP: Execution Test Results

GCClassic #9d56ba3 GEOS-Chem submod update: Merge PR #1884 (Updates for GEOS)
GEOS-Chem #b7eb4ff5d Merge PR #1891 (Fix incorrect time slice for nested-grid BC's)
HEMCO     #477c7e8 Merge PR #229 (Fix incorrect longitude definitions for HEMCO standalone)

Number of execution tests: 5

Submitted as SLURM job: 64710207
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gchp_merra2_fullchem................................Execute Simulation....PASS
gchp_merra2_fullchem_benchmark......................Execute Simulation....PASS
gchp_merra2_fullchem_RRTMG..........................Execute Simulation....PASS
gchp_merra2_tagO3...................................Execute Simulation....PASS
gchp_merra2_TransportTracers........................Execute Simulation....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 5
Execution tests failed: 0
Execution tests not yet completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Also, all integration tests were zero-diff w/r/t PR #1884 and HEMCO PR geoschem/HEMCO#229

Checking gchp_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_fullchem_benchmark
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_fullchem_RRTMG
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_tagO3
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_TransportTracers
   -> No differences in OutputDir
   -> No differences in Restarts