"Too many nested brackets" error in HEMCO 3.8.0

[yantosca]

Name and Institution (Required)

Name: Bob Yantosca
Institution: Harvard + GCST

Confirm you have reviewed the following documentation

• Support guidelines
• User manuals
• Debugging GEOS-Chem and HEMCO errors
• Current and past Github issues

Description of your issue or question

While testing PR geoschem/geos-chem#2171, I got this error in integration testing:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%               HEMCO: Harmonized Emissions Component                 %%%%%
%%%%%               You are using HEMCO version 3.8.0                     %%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  
Reading settings & switches of HEMCO configuration file: HEMCO_Config.rc

HEMCO verbose output is OFF
  
Reading fields of HEMCO configuration file: HEMCO_Config.rc
 
HEMCO ERROR: Too many nested brackets
 --> LOCATION: BracketCheck (hco_config_mod.F90)
 
HEMCO ERROR: Bracket error in HEMCO_Config.rc @ line (((.not.CEDS_01x01_SHIP
 --> LOCATION: Config_ReadCont (hco_config_mod.F90)
 
HEMCO ERROR: Error in HEMCO_Config.rc @ section: ### BEGIN SECTION BASE EMISSIONS
 --> LOCATION: Config_ReadFile (hco_config_mod.F90)

This is caused by the MAXBRACKET variable being set to 5 in HEMCO/src/Core/hco_config_mod.F90:

    ! Maximum number of nested brackets
    INTEGER, PARAMETER            :: MAXBRACKNEST = 5

I propose increasing this from 5 to a larger number such as 10. We may run into this in the future given that HEMCO does not recognize the .and. statement, but just .or. and .not..

[lizziel]

I thought that there is a limit to the number of nested brackets to avoid adding in excessive number of error checks, which can happen if the error check is deep in a calling stack. Before increasing the bracket we should probably figure out which error check is triggering the problem and make sure it is really needed. If it is deep in a series of subroutine calls then it is more likely be called in a loop.

[yantosca]

Thanks @lizziel. FWIW, the issue is happening here (see https://github.com/geoschem/geos-chem/blob/feature/ceds-0.1-degree/run/GCClassic/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem).

(((CEDS_GBDMAPS_byFuelType
(((.not.CEDS_GBDMAPS
(((.not.CEDSv2
(((.not.CEDS_01x01
>>>include $ROOT/CEDS/v2020-08/HEMCO_Config.CEDS_GBDMAPS_byFuelType.rc
))).not.CEDS_01x01
))).not.CEDSv2
))).not.CEDS_GBDMAPS
)))CEDS_GBDMAPS_byFuelType

[yantosca]

Also in HEMCO the code that checks the brackets is here:

• HEMCO/src/Core/hco_config_mod.F90

  Lines 1311 to 1558 in f807e1a

  	!------------------------------------------------------------------------------
  	! Harmonized Emissions Component (HEMCO) !
  	!------------------------------------------------------------------------------
  	!BOP
  	!
  	! !IROUTINE: BracketCheck
  	!
  	! !DESCRIPTION: Subroutine BracketCheck checks if base emission data is within
  	! a bracket and if that field shall be ignored or not. Brackets can be used to
  	! lump entires of the HEMCO configuration file into collections that can be
  	! collectively enabled or disabled. The first entry of a collection is marked
  	! adding an 'opening bracket' to the HEMCO configuration file (on the line
  	! above the entry). Opening brackets must start with three opening brackets,
  	! e.g.: '(((TEST'. Similarly, the end of a collection is marked by placing a
  	! closing bracket after the last entry of the collection: '))))TEST'.
  	! Brackets can be enabled / disabled in the EXTENSION SWITCH section of the
  	! HEMCO configuration file:
  	! \# ExtNr ExtName on/off Species
  	! 0 Base : on *
  	! --> TEST : true
  	!\\
  	!\\
  	! It is also possible to use 'opposite' brackets, e.g. to use a collection
  	! only if the given setting is *disabled*. This can be achieved by precede
  	! the collection word with '.not.', e.g. '(((.not.TEST' and '))).not.TEST'.
  	! Similarly, multiple collections can be combined to be evaluated together,
  	! e.g. NAME1.or.NAME2.
  	!\\
  	!\\
  	! !INTERFACE:
  	!
  	SUBROUTINE BracketCheck( HcoConfig, STAT, LINE, SKIP, RC )
  	!
  	! !USES:
  	!
  	USE HCO_EXTLIST_MOD, ONLY : GetExtOpt, GetExtNr
  	!
  	! !INPUT PARAMETERS:
  	!
  	INTEGER, INTENT(IN) :: STAT !
  	CHARACTER(LEN=*), INTENT(IN) :: LINE !
  	!
  	! !INPUT/OUTPUT PARAMETERS:
  	!
  	TYPE(ConfigObj), POINTER :: HcoConfig ! Config object
  	LOGICAL, INTENT(INOUT) :: SKIP ! Skip
  	INTEGER, INTENT(INOUT) :: RC ! Success/failure
  	!
  	! !REVISION HISTORY:
  	! 15 Feb 2015 - C. Keller - Initial version.
  	! See https://github.com/geoschem/hemco for complete history
  	!EOP
  	!------------------------------------------------------------------------------
  	!BOC
  	!
  	! !LOCAL VARIABLES:
  	!
  	! Maximum number of nested brackets
  	INTEGER, PARAMETER :: MAXBRACKNEST = 5
  	INTEGER :: IDX, STRLEN, ExtNr
  	LOGICAL :: FOUND
  	LOGICAL :: UseBracket, UseThis
  	LOGICAL :: verb
  	LOGICAL :: REV
  	INTEGER, SAVE :: NEST = 0
  	INTEGER, SAVE :: SKIPLEVEL = 0
  	CHARACTER(LEN=255), SAVE :: AllBrackets(MAXBRACKNEST) = ''
  	CHARACTER(LEN=255) :: TmpBracket, CheckBracket, ThisBracket
  	CHARACTER(LEN=512) :: msg
  	CHARACTER(LEN=255), PARAMETER :: LOC = 'BracketCheck (hco_config_mod.F90)'
  	!======================================================================
  	! BracketCheck begins here
  	!======================================================================
  	! Init
  	verb = HCO_IsVerb( HcoConfig%Err )
  	! Get name of this bracket
  	IF ( STAT == 5 .OR. STAT == 6 ) THEN
  	STRLEN = LEN(LINE)
  	IF ( STRLEN < 4 ) THEN
  	msg = 'Illegal bracket length: ' // TRIM(line)
  	CALL HCO_ERROR ( msg, RC, thisLoc=loc )
  	RETURN
  	ELSE
  	TmpBracket = TRIM(LINE(4:STRLEN))
  	ENDIF
  	ENDIF
  	! Open a bracket. Save out the bracket name in the list of all
  	! opened brackets. This is primarily to ensure that every opening
  	! brackets is properly closed. Only register it as skipping bracket
  	! if needed.
  	IF ( STAT == 5 ) THEN
  	! Archive bracket name
  	NEST = NEST + 1
  	IF ( NEST > MAXBRACKNEST ) THEN
  	MSG = 'Too many nested brackets'
  	CALL HCO_Error( msg, RC, thisLoc=LOC )
  	RETURN
  	ENDIF
  	AllBrackets(NEST) = TmpBracket
  	! Check if this bracket content shall be skipped. Always skip
  	! if this is a nested bracket in an already skipped bracket.
  	IF ( .NOT. SKIP ) THEN
  	! Check for 'inverse' bracket. These start with '.not.'
  	CheckBracket = TmpBracket
  	REV = .FALSE.
  	IF ( STRLEN > 5 ) THEN
  	IF ( TmpBracket(1:5) == '.not.' ) THEN
  	STRLEN = LEN(TmpBracket)
  	CheckBracket = TmpBracket(6:STRLEN)
  	REV = .TRUE.
  	ENDIF
  	ENDIF
  	! Check if the evaluation of CheckBracket returns true, i.e.
  	! if any of the elements of CheckBracket is enabled. These
  	! can be multiple settings separated by '.or.'.
  	! By default, don't use the content of the bracket
  	UseBracket = .FALSE.
  	! Make sure variable ThisBracket is initialized. Needed in the
  	! DO loop below
  	ThisBracket = ''
  	! Pack the following into a DO loop to check for multiple
  	! flags separated by '.or.'.
  	DO
  	! Leave do loop if ThisBracket is equal to CheckBracket.
  	! In this case, the entire bracket has already been
  	! evaluated.
  	IF ( TRIM(CheckBracket) == TRIM(ThisBracket) ) EXIT
  	! Evaluate bracket for '.or.':
  	IDX = INDEX(TRIM(CheckBracket),'.or.')
  	! If '.or.' is a substring of the whole bracket, get
  	! substring up to the first '.or.' and write it into variable
  	! ThisBracket, which will be evaluated below. The tail
  	! (everything after the first '.or.') is written into
  	! CheckBracket.
  	IF ( IDX > 0 ) THEN
  	ThisBracket = CheckBracket(1:(IDX-1))
  	STRLEN = LEN(CheckBracket)
  	CheckBracket = CheckBracket((IDX+4):STRLEN)
  	! If there is no '.or.' in the bracket, simply evaluate the
  	! whole bracket.
  	ELSE
  	ThisBracket = CheckBracket
  	ENDIF
  	! Check if this bracket has been registered as being used.
  	! Scan all extensions, including the core one.
  	CALL GetExtOpt( HcoConfig, -999, TRIM(ThisBracket), &
  	OptValBool=UseThis, FOUND=FOUND, RC=RC )
  	IF ( RC /= HCO_SUCCESS ) THEN
  	msg = 'Error when checking '// TRIM( thisBracket ) // '!'
  	CALL HCO_Error( msg, RC, thisLoc=loc )
  	RETURN
  	ENDIF
  	! If bracket name was found in options, update the UseBracket
  	! variable accordingly.
  	IF ( FOUND ) THEN
  	UseBracket = UseThis
  	! If bracket name was not found, check if this is an extension
  	! name
  	ELSE
  	ExtNr = GetExtNr( HcoConfig%ExtList, TRIM(ThisBracket) )
  	IF ( ExtNr > 0 ) THEN
  	UseBracket = .TRUE.
  	ENDIF
  	ENDIF
  	! As soon as UseBracket is true, we don't need to evaluate
  	! further
  	IF ( UseBracket ) EXIT
  	ENDDO
  	! We need to skip the content of this bracket?
  	SKIP = .NOT. UseBracket
  	! Eventually reverse the skip flag
  	IF ( REV ) THEN
  	SKIP = .NOT. SKIP
  	ENDIF
  	! If bracket is skipped, adjust skip level accordingly.
  	! This is so that we know when it's time to flip back to
  	! a bracket that is being used (if brackets are nested).
  	IF ( SKIP ) THEN
  	SKIPLEVEL = NEST
  	ENDIF
  	ENDIF
  	! Verbose mode
  	IF ( verb ) THEN
  	MSG = 'Opened shortcut bracket: '//TRIM(TmpBracket)
  	CALL HCO_MSG( HcoConfig%Err, msg )
  	WRITE(MSG,*) ' - Skip content of this bracket: ', SKIP
  	CALL HCO_MSG( HcoConfig%Err, msg )
  	ENDIF
  	ENDIF
  	! Close a bracket
  	IF ( STAT == 6 ) THEN
  	! This must be the latest opened bracket
  	IF ( TRIM(TmpBracket) /= TRIM(AllBrackets(NEST)) ) THEN
  	MSG = 'Closing bracket does not match opening bracket: '// &
  	TRIM(TmpBracket)//', expected: '//TRIM(AllBrackets(NEST))
  	CALL HCO_Error( msg, RC, thisLoc=LOC )
  	RETURN
  	ENDIF
  	! If that was the latest opened bracket that was disabled
  	IF ( SKIPLEVEL == NEST ) THEN
  	SKIP = .FALSE.
  	ENDIF
  	! Update nesting level
  	AllBrackets(NEST) = ''
  	NEST = NEST - 1
  	! Verbose mode
  	IF ( verb ) THEN
  	MSG = 'Closed shortcut bracket: '//TRIM(TmpBracket)
  	CALL HCO_MSG( HcoConfig%Err, msg )
  	WRITE(MSG,*) ' - Skip following lines: ', SKIP
  	CALL HCO_MSG( HcoConfig%Err, msg )
  	ENDIF
  	ENDIF
  	! Return w/ success
  	RC = HCO_SUCCESS
  	END SUBROUTINE BracketCheck

[msulprizio]

Is there a reason to keep CEDSv2 (i.e. the coarse-resolution version) in geoschem/geos-chem#2171? That could help reduce the nesting in HEMCO_Config.rc?

[yantosca]

I could take it out @msulprizio. I wasn't sure if there's still a need for it.

[yantosca]

@msulprizio: the CEDSv2 data goes back to 1750 but the CEDS_01x01 goes back to 1980. I wasn't sure if people needed the historical data, which is why I thought to keep both.

[lizziel]

Ah, I am confusing this with a different max number of nests set in HEMCO.

[msulprizio]

@msulprizio: the CEDSv2 data goes back to 1750 but the CEDS_01x01 goes back to 1980. I wasn't sure if people needed the historical data, which is why I thought to keep both.

Thanks. I've commented on the original pull request and tagged the developers to see if we can reduce the number of CEDS options (and therefore nested brackets) in the default HEMCO_Config.rc.

[yantosca]

Thanks @msulprizio @lizzie. The reason we were hitting the max bracket was because of all the CEDS options.

[yantosca]

I thought that there is a limit to the number of nested brackets to avoid adding in excessive number of error checks, which can happen if the error check is deep in a calling stack. Before increasing the bracket we should probably figure out which error check is triggering the problem and make sure it is really needed. If it is deep in a series of subroutine calls then it is more likely be called in a loop.

@lizziel: I believe this is done in the initialization stage, when the config file is read. The end result is that a list of logicals is created that indicate if each bracket is activated or deactivated. So I think it doesn't add too much more overhead.

[yantosca]

We can now close this issue as PR #262 has been merged into the HEMCO "no-diff-to-benchmark" development stream.