Add prototype HcalNanoAOD

[DryRun]

PR description:

This PR is a continuation of #40489, which was mistakenly closed after a rebase.

This PR implements a NanoAOD format for HCAL digis and RecHits. This data format is intended to replace the old HcalTupleMaker used by most HCAL workflows.

Example:

cmsDriver.py NANO \
    -s RAW2DIGI,RECO,USER:DPGAnalysis/HcalNanoAOD/hcalNano_cff.hcalNanoTask \
    --datatier NANOAOD \
    --eventcontent NANOAOD \
    --filein root://eoscms.cern.ch//store/group/dpg_hcal/comm_hcal/RAW/ZeroBias/361579/be9a0a12-548c-4aa0-837a-2d5654f8e63b.root \
    --fileout hcalnano_testcmsdriver.root \
    -n 100 \
    --nThreads 4 \
    --conditions auto:run3_data_prompt \
    --era Run3 \
    --python_filename testcmsdriver_cfg.py \
    --no_exec
cmsRun testcmsdriver_cfg.py

RecHits are implemented following @kdlong's PFNano branch, https://github.com/kdlong/cmssw/tree/pfNano_HCALOnly_12_6_0_pre4. The specific modules, however, are defined in DPGAnalysis/HcalNanoAOD (e.g. HBHERecHitFlatTableProducer), rather than DPGAnalysis/CaloNanoAOD (e.g. SimpleCaloRecHitFlatTableProducer). Hence, this package is independent of DPGAnalysis/CaloNanoAOD.

Digis are handled by dedicated modules in DPGAnalysis/HcalNanoAOD that output dense arrays, sorted by DetId. A list of the DetIds (i.e. the column labels) is placed in Run. For the moment, DetIds are also saved in Event, for development purposes although we might decide to keep them for ease of analysis.

Metadata from the uHTRs (calib event type) and the uMNio (the labeling of calibration events (pedestal/LED/laser) and which type of laser event), are also handled by dedicated modules.

Two customization functions are provided for special datasets:

1. hcalNano_cff.customiseHcalCalib: for calibration sequence events in the TestEnablesEcalHcal dataset (i.e. events in the abort gap of type pedestal, LED, or laser). Specifies the correct label for the raw data; adds uMNio metadata; adds an important HLT filter to all Paths and Endpaths, which selects events triggered by HLT_HcalCalibration. This filter essential to avoid a crash, because ECAL-triggered events are missing the HCAL raw data collection (aside: HCAL needs to revisit the definition of this dataset; the trigger is based on a majority-wins vote of calibration flags from the HCAL uHTRs. A single uHTR flagging the event as calibration should be enough to consider the event calibration and not real physics!).
2. hcalNano_cff.customiseHcalLocal: for HCAL local runs. Loads the raw data using HcalTBSource; adjusts the number of time slices per digi; adds uMNio metadata.

TBD: trigger primitives.

PR validation:

Produces useful HCAL NanoAOD ntuples. Data validated event-by-event against HcalTupleMaker. Calibration and local run customization functions work on recent MWGR 2023 data.

Other

The contents of this PR are ready for use by HCAL DPG right away. An eventual merge should take into account @kdlong's PFNanoAOD branch, but this package is independent of DPGAnalysis/CaloNanoAOD.

@vlimant @swertz this should be assigned to XPOG again.

@mariadalfonso @wang-hui @jhakala

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41607/35499

• This PR adds an extra 60KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): remove 2022 HLT menu from CMSSW_13_X_Y #41471, Introduce variant of pp_on_PbPb era for 2023 #41571

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41607/35499/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41607/35499/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41607/35508

• This PR adds an extra 76KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): remove 2022 HLT menu from CMSSW_13_X_Y #41471, Introduce variant of pp_on_PbPb era for 2023 #41571

[cmsbuild]

A new Pull Request was created by @DryRun (David Yu) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)
• DPGAnalysis/HcalNanoAOD (****)
• DataFormats/HcalDetId (simulation)

The following packages do not have a category, yet:

DPGAnalysis/HcalNanoAOD
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@civanch, @bbilin, @mdhildreth, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@missirol, @makortel, @kpedro88, @Martin-Grunewald, @bsunanda, @rovere, @mariadalfonso, @abdoulline, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[srimanob]

@DryRun Is this PR ready to review?

[DryRun]

Hi @srimanob , yes, the PR is ready for review.

[civanch]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b6d266/32731/summary.html
COMMIT: 7b13a66
CMSSW: CMSSW_13_2_X_2023-05-20-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41607/32731/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 15 lines from the logs
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3221591
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3221569
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 207 log files, 159 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b6d266/33309/summary.html
COMMIT: fc2f7a6
CMSSW: CMSSW_13_2_X_2023-06-21-1600/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41607/33309/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
25034.114 step 2
25034.21 step 2
25034.9921 step 2
25034.999 step 2
25034.99 step 2
25061.97 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially removed 6 lines from the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3200270
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3200242
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 207 log files, 159 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[srimanob]

+Upgrade

For upgrade side on configuration, new workflows run fine.

[DryRun]

The outputs from the matrix tests 1060.1 and 1060.2 look good.

[srimanob]

Kindly ping @cms-sw/pdmv-l2 @cms-sw/xpog-l2 @cms-sw/simulation-l2

[sunilUIET]

+pdmv

[vlimant]

+1
we will address the cmsDriver syntax and workflow location in a follow up PR

[civanch]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1