0.001 x 0.001 degree masks cause performance slowdown in GCHP for carbon simulations

[yantosca]

Name and Institution (Required)

Name: Bob Yantosca
Institution: Harvard + GCST

Confirm you have reviewed the following documentation

• Contributing guidelines

Overview

In the GCHP carbon simulation, the Scarpelli_Canada and Scarpelli_Mexico emissions use these mask files described below.. @lizziel found that reading/regridding these mask files causes a substantial performance penalty in GCHP. This may also cause a slowdown in GCClassic CH4 and carbon simulations (including inversions for the IMI).

Specification in HEMCO_Config.rc:

1001 MEX_MASK          $ROOT/MASKS/v2024-01/Mexico_Mask.001x001.nc        MASK 2000/1/1/0 C xy 1 1 -118/17/-95/33
1010 MEX_MASK_MIRROR   $ROOT/MASKS/v2024-01/Mexico_Mask_Mirror.001x001.nc MASK 2000/1/1/0 C xy 1 1 -118/17/-95/33
1002 CAN_MASK          $ROOT/MASKS/v2024-01/Canada_Mask.001x001.nc        MASK 2000/1/1/0 C xy 1 1 -141/40/-52/85
1011 CAN_MASK_MIRROR   $ROOT/MASKS/v2024-01/Canada_Mask_Mirror.001x001.nc MASK 2000/1/1/0 C xy 1 1 -141/40/-52/85

Specification in ExtData.rc

MEX_MASK          1 N Y - none none MASK ./HcoDir/MASKS/v2024-01/Mexico_Mask.001x001.nc
MEX_MASK_MIRROR   1 N Y - none none MASK ./HcoDir/MASKS/v2024-01/Mexico_Mask_Mirror.001x001.nc
CAN_MASK          1 N Y - none none MASK ./HcoDir/MASKS/v2024-01/Canada_Mask.001x001.nc
CAN_MASK_MIRROR   1 N Y - none none MASK ./HcoDir/MASKS/v2024-01/Canada_Mask_Mirror.001x001.nc

Mask files
The mask files are at high horizontal resolution (0.001 x 0.001 degree) and include a subsetted area around the region of interest (Canada or Mexico). The mask coverages are shown below. NOTE: The mirror masks images are omitted, they are just the inverse of the original masks (i.e. NOT Mexico and NOT Canada).

Proposed solution
Creating smaller mask files will reduce the computational overhead involved in reading/regridding. Both the Scarpelli_Canada and Scarpelli_Mexico emissions are 0.1 x 0.1 degrees, so we shouldn't need mask files with resolution finer than 0.1 x 0.1 resolution.

Related issues

• GCHP carbon simulations with CH4 take much longer than simulations without CH4 GCHP#411

Tagging @msulprizio @lizziel @laestrada

[yantosca]

Also forgot to tag @kbowman77 and @hannahnesser

[yantosca]

Timing results (1-hour carbon simulation, all species)

Prior to PR #2272, the ExtData component took in excess of 2000 seconds:

                                                               Inclusive        Exclusive    
                                                            ================ ================
Name                                               #-cycles  T (sec)    %     T (sec)    %   
                                                   -------- --------- ------ --------- ------
All                                                       1  2061.359 100.00     0.095   0.00
--SetService                                              1     0.180   0.01     0.005   0.00
----GCHP                                                  1     0.175   0.01     0.035   0.00
------GCHPctmEnv                                          1     0.002   0.00     0.002   0.00
------GCHPchem                                            1     0.065   0.00     0.065   0.00
------DYNAMICS                                            1     0.073   0.00     0.073   0.00
----HIST                                                  1     0.000   0.00     0.000   0.00
----EXTDATA                                               1     0.000   0.00     0.000   0.00
--Initialize                                              1    15.827   0.77     0.100   0.00
----GCHP                                                  1     5.393   0.26     0.143   0.01
------GCHPctmEnv                                          1     0.008   0.00     0.008   0.00
------GCHPchem                                            1     5.236   0.25     5.236   0.25
------DYNAMICS                                            1     0.006   0.00     0.006   0.00
----HIST                                                  1     0.165   0.01     0.165   0.01
----EXTDATA                                               1    10.169   0.49    10.169   0.49
--Run                                                     1  2045.026  99.21     0.076   0.00
----EXTDATA                                               6  2043.660  99.14  2043.660  99.14
----GCHP                                                 12     0.975   0.05     0.001   0.00
------GCHPctmEnv                                         12     0.179   0.01     0.179   0.01
------GCHPchem                                           12     0.520   0.03     0.520   0.03
------DYNAMICS                                           12     0.275   0.01     0.275   0.01
----HIST                                                 12     0.314   0.02     0.314   0.02
--Finalize                                                1     0.231   0.01     0.036   0.00
----GCHP                                                  1     0.193   0.01     0.001   0.00
------GCHPctmEnv                                          1     0.001   0.00     0.001   0.00
------GCHPchem                                            1     0.189   0.01     0.189   0.01
------DYNAMICS                                            1     0.002   0.00     0.002   0.00
----HIST                                                  1     0.001   0.00     0.001   0.00
----EXTDATA                                               1     0.002   0.00     0.002   0.00

After implementing PR #2272, the ExtData component took less than 70 seconds for a 1-hour carbon simulation (with all species).

                                                               Inclusive        Exclusive    
                                                            ================ ================
Name                                               #-cycles  T (sec)    %     T (sec)    %   
                                                   -------- --------- ------ --------- ------
All                                                       1    83.281 100.00     0.046   0.05
--SetService                                              1     0.174   0.21     0.005   0.01
----GCHP                                                  1     0.170   0.20     0.028   0.03
------GCHPctmEnv                                          1     0.002   0.00     0.002   0.00
------GCHPchem                                            1     0.065   0.08     0.065   0.08
------DYNAMICS                                            1     0.075   0.09     0.075   0.09
----HIST                                                  1     0.000   0.00     0.000   0.00
----EXTDATA                                               1     0.000   0.00     0.000   0.00
--Initialize                                              1    13.716  16.47     0.078   0.09
----GCHP                                                  1     4.799   5.76     0.116   0.14
------GCHPctmEnv                                          1     0.007   0.01     0.007   0.01
------GCHPchem                                            1     4.646   5.58     4.646   5.58
------DYNAMICS                                            1     0.029   0.04     0.029   0.04
----HIST                                                  1     0.165   0.20     0.165   0.20
----EXTDATA                                               1     8.675  10.42     8.675  10.42
--Run                                                     1    69.153  83.04     0.018   0.02
----EXTDATA                                               6    67.819  81.43    67.819  81.43
----GCHP                                                 12     0.993   1.19     0.001   0.00
------GCHPctmEnv                                         12     0.183   0.22     0.183   0.22
------GCHPchem                                           12     0.525   0.63     0.525   0.63
------DYNAMICS                                           12     0.283   0.34     0.283   0.34
----HIST                                                 12     0.324   0.39     0.324   0.39
--Finalize                                                1     0.191   0.23     0.003   0.00
----GCHP                                                  1     0.185   0.22     0.001   0.00
------GCHPctmEnv                                          1     0.001   0.00     0.001   0.00
------GCHPchem                                            1     0.181   0.22     0.181   0.22
------DYNAMICS                                            1     0.002   0.00     0.002   0.00
----HIST                                                  1     0.001   0.00     0.001   0.00
----EXTDATA                                               1     0.002   0.00     0.002   0.00

Maximum memory usage (1-hour carbon simulation, all species)

Prior to PR #2272:

Mem/Swap Used (MB) at MAPL_Cap:TimeLoop=  1.603E+05  0.000E+00

After implementing PR #2272

Mem/Swap Used (MB) at MAPL_Cap:TimeLoop=  5.128E+04  0.000E+00

Tagging @lizziel @msulprizio

[msulprizio]

@yantosca That is a fantastic improvement!

[yantosca]

GEOS-Chem Classic timing information

Prior to PR #2272:

===============================================================================
G E O S - C H E M   T I M E R S
 
  Timer name                       DD-hh:mm:ss.SSS     Total Seconds
-------------------------------------------------------------------------------
  GEOS-Chem                     :  00-00:01:32.875            92.875
  HEMCO                         :  00-00:01:19.750            79.750
  All chemistry                 :  00-00:00:02.125             2.125
  => Gas-phase chem             :  00-00:00:02.125             2.125
  => Photolysis                 :  >>>>> THE TIMER DID NOT RUN <<<<<
  => Aerosol chem               :  >>>>> THE TIMER DID NOT RUN <<<<<
  => Linearized chem            :  >>>>> THE TIMER DID NOT RUN <<<<<
  Transport                     :  00-00:00:05.000             5.000
  Convection                    :  00-00:00:00.250             0.250
  Boundary layer mixing         :  00-00:00:02.250             2.250
  Dry deposition                :  >>>>> THE TIMER DID NOT RUN <<<<<
  Wet deposition                :  >>>>> THE TIMER DID NOT RUN <<<<<
  Diagnostics                   :  00-00:00:01.500             1.500
  Unit conversions              :  00-00:00:00.125             0.125

After implementing the fix in PR #2272:

===============================================================================
G E O S - C H E M   T I M E R S
 
  Timer name                       DD-hh:mm:ss.SSS     Total Seconds
-------------------------------------------------------------------------------
  GEOS-Chem                     :  00-00:01:28.500            88.500
  HEMCO                         :  00-00:01:15.250            75.250
  All chemistry                 :  00-00:00:02.250             2.250
  => Gas-phase chem             :  00-00:00:02.250             2.250
  => Photolysis                 :  >>>>> THE TIMER DID NOT RUN <<<<<
  => Aerosol chem               :  >>>>> THE TIMER DID NOT RUN <<<<<
  => Linearized chem            :  >>>>> THE TIMER DID NOT RUN <<<<<
  Transport                     :  00-00:00:04.750             4.750
  Convection                    :  00-00:00:00.500             0.500
  Boundary layer mixing         :  00-00:00:02.250             2.250
  Dry deposition                :  >>>>> THE TIMER DID NOT RUN <<<<<
  Wet deposition                :  >>>>> THE TIMER DID NOT RUN <<<<<
  Diagnostics                   :  00-00:00:01.625             1.625
  Unit conversions              :  00-00:00:00.125             0.125

[kbowman77]

Is this going to be an issue for 11km or finer runs?

[yantosca]

@kbowman77: I don't think it will be.

[yantosca]

We can now close this issue as PR #2272 has been merged into the GEOS-Chem 14.4.0 development stream.