Definition of a new DQM Offline sequence to be run on Express Stream

[rvenditti]

PR description:

The goal of this PR is to define a new sequence in DQM Offline, that should be run on the Express Stream. The new sequence will produce a reduced number of ME (10% less) wrt the DQM sequence that is presently run on Express (the "standardDQM" sequence).
By this way, also the number of ME that should be harvested in step3 will be reduced (thanks to a new ad-hoc harvesting sequence), and hopefully this would mitigate the memory issues observed at T0 during 2022 and 2023 collisions (discussed here #38976).

The subsystem sequences that will be run with this new configuration, are decided on the basis of the feedback received during the DQM General meeting https://indico.cern.ch/event/1259866/. Wrt the old sequence, we removed DT, EGamma, HCAL, JetMet, L1T and Physics. Additional room for improvement: remove MuonPOG (not yet done, since part of the CSC plots are produced in the MuonPOG sequence).

PR validation:

A generic test that mimic the offline reconstruction+ DQM steps on top of the raw express data has been performed with

cmsDriver.py step2 --conditions 130X_dataRun3_Express_v2 --data --datatier RECO,DQMIO --era Run3 --eventcontent RECO,DQM --filein file:/eos/cms/store/t0streamer/Data/Express/000/368/454/run368454_ls0134_streamExpress_StorageManager.dat --fileout file:step2_modified.root --nStreams 2 --nThreads 8 --no_exec --number 10 --process RECO,DQM --python_filename step2_modified_cfg.py --scenario pp --step RAW2DIGI,L1Reco,RECO,DQM:@standardDQMExpress

cmsDriver.py step3 --conditions 130X_dataRun3_Express_v2 --data --era Run3 --filein step2_modified_inDQM.root --fileout file:step3_modified.root --filetype DQM --nStreams 2 --no_exec --number 10 --python_filename step_3_modified_cfg.py --scenario pp --step HARVESTING:@standardDQMExpress

The DQMIO files produced from the old (standardDQM) and new (standardDQMExpress) workflow have been inspected with dqmiodumpmetadata.py and dqmiolistmes.py scripts.

This PR should be backported to 13_0_X, because we target to have the new sequence running at T0.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41944/35913

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @rvenditti for master.

It involves the following packages:

• Configuration/StandardSequences (operations)
• DQMOffline/Configuration (dqm)

@perrotta, @cmsbuild, @rappoccio, @nothingface0, @emanueleusai, @davidlange6, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @fabiocos, @rvenditti can you please review it and eventually sign? Thanks.
@fabiocos, @VourMa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @mtosi, @mmusich, @sameasy, @threus, @dgulhan, @slomeo, @rociovilar this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[emanueleusai]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f2e61/33134/summary.html
COMMIT: 9420ae9
CMSSW: CMSSW_13_2_X_2023-06-13-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41944/33134/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
25034.114 step 2
25034.21 step 2
25034.9921 step 2
25034.999 step 2
25034.99 step 2
25061.97 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially added 8 lines to the logs
• Reco comparison results: 9 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3190910
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3190882
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 207 log files, 159 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[emanueleusai]

+1

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.