Improve L1TMuonBarrelTrackProducer #46640

Dr15Jones · 2024-11-08T21:52:28Z

PR description:

decreased substantially the number of per event allocations done
made code changes to allow this module to now be a stream module
improved memory handling

PR validation:

Ran step3 of workflow 12861.0 with improved ModuleAllocMonitor. Previously each event processed by the module needed 85k allocations. Now only the first event of each stream uses that much and all subsequent only use 19 allocations.

This should strictly be a technical change and should have no effect on the results.

- Hold member data directly not via pointers - Changed interface to reflect data is always available - Fixed const correctness issues - Removed unnecessary virtual interfaces

This required moving the header file to follow CMS coding rules.

Moved producer header info into source.

This routine was found to do an excessive number of allocations.

cmsbuild · 2024-11-08T21:52:55Z

cms-bot internal usage

Dr15Jones · 2024-11-08T21:52:59Z

@makortel FYI

Dr15Jones · 2024-11-08T21:53:08Z

please test

Dr15Jones · 2024-11-08T21:54:35Z

The reason the allocations dropped is the code now only calls L1MuBMTFConfig::setDefaultsES if the dependent record has changed.

cmsbuild · 2024-11-08T21:54:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46640/42551

cmsbuild · 2024-11-08T21:54:59Z

A new Pull Request was created by @Dr15Jones for master.

It involves the following packages:

L1Trigger/L1TMuonBarrel (l1)

@aloeliger, @epalencia can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @dinyar, @eyigitba, @missirol, @mmusich, @thomreis this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

makortel · 2024-11-08T22:32:23Z

type performance-improvements

cmsbuild · 2024-11-08T23:37:27Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Size: This PR adds an extra 216KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8e66cc/42691/summary.html
COMMIT: ee26979
CMSSW: CMSSW_14_2_X_2024-11-08-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46640/42691/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

136.793

----- Begin Fatal Exception 08-Nov-2024 23:12:30 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 301998 lumi: 9 event: 8792128 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'packGmtStage2'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

136.731

----- Begin Fatal Exception 08-Nov-2024 23:12:31 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 274199 lumi: 21 event: 39389360 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'packGmtStage2'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 08-Nov-2024 23:12:32 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 105676750 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'packGmtStage2'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

136.724136.724_RunJetHT2016B/step2_RunJetHT2016B.log
136.731136.731_RunSinglePh2016B/step2_RunSinglePh2016B.log
136.735136.735_RunNoBPTX2016B/step2_RunNoBPTX2016B.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_Fake2

----- Begin Fatal Exception 08-Nov-2024 23:14:06 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 3
   [1] Running path 'RAWSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'RAWSIMoutput'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'gmtStage2Raw'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

hlt_data_Fake2

----- Begin Fatal Exception 08-Nov-2024 23:12:44 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 272762 lumi: 51 event: 41555597 stream: 0
   [1] Running path 'RAWoutput_step'
   [2] Prefetching for module PoolOutputModule/'RAWoutput'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'packGmtStage2'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

Dr15Jones · 2024-11-09T14:45:27Z

I was able to reproduce the problem locally and then used git bisect to determine which commit introduced a problem. It was

ee26979 is the first bad commit

Author: Christopher Jones <[email protected]>
Date:   Fri Nov 8 14:57:31 2024 -0600

    Make L1TMuonBarrelTrackProducer a stream module
    
    No more global variables are being used by code called by the
    producer.
    Switched to using edm::Event::emplace.

cmsbuild · 2024-11-12T22:43:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46640/42623

cmsbuild · 2024-11-12T22:43:33Z

Pull request #46640 was updated. @aloeliger, @atpathak, @cmsbuild, @consuegs, @epalencia, @francescobrivio, @perrotta can you please check and sign again.

Dr15Jones · 2024-11-12T22:45:16Z

I switched on Debug output from the module and saw that the building stage was not occurring.

Dr15Jones · 2024-11-12T22:54:30Z

please test

cmsbuild · 2024-11-13T03:01:02Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8e66cc/42779/summary.html
COMMIT: 5975fee
CMSSW: CMSSW_14_2_X_2024-11-12-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46640/42779/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
2024.101001 step 1
2024.202001 step 1
2024.303001 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 3 lines from the logs
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3070351
DQMHistoTests: Total failures: 395
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3069936
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 190 log files, 160 edm output root files, 43 DQM output files
TriggerResults: no differences found

perrotta · 2024-11-13T07:34:52Z

+1

aloeliger · 2024-11-13T22:59:43Z

+l1

cmsbuild · 2024-11-13T23:00:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @mandrenguyen, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2024-11-15T09:54:43Z

+1

Dr15Jones added 6 commits November 8, 2024 10:18

Improve L1MuBMSectorProcessor memory handling

c54e7eb

- Hold member data directly not via pointers - Changed interface to reflect data is always available - Fixed const correctness issues - Removed unnecessary virtual interfaces

Get L1MuBMTFConfig from L1MuBMTrackFinder

ad53dce

Move L1MuBMTFConfig into L1MuBMTrackFinder

c7942de

This required moving the header file to follow CMS coding rules.

Better memory handling in L1MuBMTrackFinder

b240478

Improved memory handling in L1TMuonBarrelTrackProducer

d69ff33

Moved producer header info into source.

Only call setDefaultsES if record has changed

679ff5c

This routine was found to do an excessive number of allocations.

cmsbuild added this to the CMSSW_14_2_X milestone Nov 8, 2024

cmsbuild added l1-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 8, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 8, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 8, 2024

Dr15Jones mentioned this pull request Nov 8, 2024

Using neutrino gun events to probe memory growth #46598

Open

makortel mentioned this pull request Nov 8, 2024

Improve L1TMuonBarrelTrackProducer cms-sw/framework-team#1083

Closed

cmsbuild added the performance-improvements label Nov 8, 2024

cmsbuild added tests-rejected and removed tests-started labels Nov 8, 2024

cmsbuild added tests-pending code-checks-pending labels Nov 12, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 12, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 12, 2024

cmsbuild added tests-approved and removed tests-started labels Nov 13, 2024

cmsbuild added alca-approved db-approved and removed alca-pending db-pending labels Nov 13, 2024

cmsbuild added l1-approved fully-signed and removed l1-pending pending-signatures labels Nov 13, 2024

cmsbuild added orp-approved and removed orp-pending labels Nov 15, 2024

cmsbuild merged commit bdf7b42 into cms-sw:master Nov 15, 2024
11 checks passed

cmsbuild mentioned this pull request Nov 15, 2024

Remove redefinition of EDM_ML_DEBUG in HcalDigitizer.cc #46708

Merged

Dr15Jones deleted the improveL1TMuonBarrel branch November 19, 2024 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve L1TMuonBarrelTrackProducer #46640

Improve L1TMuonBarrelTrackProducer #46640

Dr15Jones commented Nov 8, 2024

cmsbuild commented Nov 8, 2024 •

edited

Loading

Dr15Jones commented Nov 8, 2024

Dr15Jones commented Nov 8, 2024

Dr15Jones commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

makortel commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

Dr15Jones commented Nov 9, 2024

cmsbuild commented Nov 12, 2024

cmsbuild commented Nov 12, 2024

Dr15Jones commented Nov 12, 2024

Dr15Jones commented Nov 12, 2024

cmsbuild commented Nov 13, 2024

perrotta commented Nov 13, 2024

aloeliger commented Nov 13, 2024

cmsbuild commented Nov 13, 2024

mandrenguyen commented Nov 15, 2024

Improve L1TMuonBarrelTrackProducer #46640

Improve L1TMuonBarrelTrackProducer #46640

Conversation

Dr15Jones commented Nov 8, 2024

PR description:

PR validation:

cmsbuild commented Nov 8, 2024 • edited Loading

Dr15Jones commented Nov 8, 2024

Dr15Jones commented Nov 8, 2024

Dr15Jones commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

makortel commented Nov 8, 2024

cmsbuild commented Nov 8, 2024

RelVals

RelVals-INPUT

AddOn Tests

Dr15Jones commented Nov 9, 2024

cmsbuild commented Nov 12, 2024

cmsbuild commented Nov 12, 2024

Dr15Jones commented Nov 12, 2024

Dr15Jones commented Nov 12, 2024

cmsbuild commented Nov 13, 2024

Comparison Summary

perrotta commented Nov 13, 2024

aloeliger commented Nov 13, 2024

cmsbuild commented Nov 13, 2024

mandrenguyen commented Nov 15, 2024

cmsbuild commented Nov 8, 2024 •

edited

Loading