UPDATED: Adding of jet purity and efficiency plots to PFValidation package #43304

peteryouBuffalo · 2023-11-16T17:59:17Z

PR description:

This pull request is a revision of a previous open pull request (#39902) which added plots for comparison of purity and reconstruction efficiency of jets. In the original PR there were extraneous plots being produced and some spacing issues; this PR fixes those and updates compatibility to the current master.
@laurenhay

PR validation:

Ran the PF validation and ensured that the new plots were being produced. Also did typical scram code checks and code format.

cmsbuild · 2023-11-16T18:06:41Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43304/37745

This PR adds an extra 40KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Validation/RecoParticleFlow/README.md modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/OffsetAnalyzerDQM.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/PFJetAnalyzerDQM.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/PFJetDQMPostProcessor.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/defaults_cfi.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/offsetAnalyzerDQM_cff.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/particleFlowDQM_cff.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/test/compare.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902

cmsbuild · 2023-11-16T18:07:06Z

A new Pull Request was created by @peteryouBuffalo for master.

It involves the following packages:

Configuration/Applications (operations)
Validation/RecoParticleFlow (dqm)

@syuvivida, @fabiocos, @davidlange6, @rappoccio, @nothingface0, @antoniovagnerini, @antoniovilela, @cmsbuild, @rvenditti, @tjavaid can you please review it and eventually sign? Thanks.
@makortel, @missirol, @fabiocos, @Martin-Grunewald, @hatakeyamak this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

swagata87 · 2023-11-18T06:32:21Z

type pf

swagata87 · 2023-11-18T06:34:22Z

@cmsbuild please test

cmsbuild · 2023-11-18T10:57:35Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d91dab/35935/summary.html
COMMIT: 0715759
CMSSW: CMSSW_14_0_X_2023-11-17-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43304/35935/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 63 lines from the logs
Reco comparison results: 15 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3363868
DQMHistoTests: Total failures: 1852
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3361994
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 802.5509999999997 KiB( 49 files compared)
DQMHistoSizes: changed ( 10024.0,... ): 24.809 KiB ParticleFlow/JetResponse
DQMHistoSizes: changed ( 10024.0,... ): 0.812 KiB ParticleFlow/Offset
DQMHistoSizes: changed ( 25.0,... ): 10.494 KiB ParticleFlow/JetResponse
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

makortel · 2023-11-20T14:29:07Z

Configuration/Applications/python/ConfigBuilder.py

@@ -16,6 +16,7 @@
 import collections
 from subprocess import Popen,PIPE
 import FWCore.ParameterSet.DictTypes as DictTypes
+from FWCore.ParameterSet.OrderedSet import OrderedSet


These changes are already in the ConfigBuilder.py.

Won't it just merge then? Let us know if we need to change anything.

In principle yes, but the PR would be cleaner, and avoid the operations signature, if the PR would be rebased to a (pre)release/IB that contains that commit. Or just drop the first commit without rebasing (it is not immediately clear why it would be needed for this PR even in an older release)

I see now! I did not notice that we had picked up that commit somehow. We will remove that commit.

@makortel done! Sorry again about that.

cmsbuild · 2023-11-20T17:57:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43304/37793

This PR adds an extra 12KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Validation/RecoParticleFlow/README.md modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/OffsetAnalyzerDQM.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/PFJetAnalyzerDQM.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/plugins/PFJetDQMPostProcessor.cc modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/defaults_cfi.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/offsetAnalyzerDQM_cff.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/python/particleFlowDQM_cff.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902
- File Validation/RecoParticleFlow/test/compare.py modified in PR(s): Adding jet purity and efficiency plots to PFValidation package #39902

cmsbuild · 2023-11-20T17:58:05Z

Pull request #43304 was updated. @antoniovagnerini, @cmsbuild, @nothingface0, @rvenditti, @tjavaid, @syuvivida can you please check and sign again.

tjavaid · 2023-11-21T03:35:32Z

please test

cmsbuild · 2023-11-21T10:02:32Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d91dab/35970/summary.html
COMMIT: 0adf85b
CMSSW: CMSSW_14_0_X_2023-11-20-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43304/35970/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 104 lines from the logs
Reco comparison results: 129 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3363868
DQMHistoTests: Total failures: 2830
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3361016
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 802.5509999999997 KiB( 49 files compared)
DQMHistoSizes: changed ( 10024.0,... ): 24.809 KiB ParticleFlow/JetResponse
DQMHistoSizes: changed ( 10024.0,... ): 0.812 KiB ParticleFlow/Offset
DQMHistoSizes: changed ( 25.0,... ): 10.494 KiB ParticleFlow/JetResponse
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

tjavaid · 2023-11-25T03:38:54Z

+1

additional differences coming from known unstable WF 12434.7 spurious differences in outputs of Run-3 wfs *.7 #39803

cmsbuild · 2023-11-25T03:39:20Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2023-11-27T15:17:36Z

Are we closing #39902 in favor of this one?

laurenhay · 2023-11-27T16:01:52Z

Are we closing #39902 in favor of this one?

Yes

rappoccio · 2023-11-28T22:17:17Z

+1

mmusich · 2023-12-02T18:42:42Z

Validation/RecoParticleFlow/plugins/PFJetDQMPostProcessor.cc

+        h_purity->Divide(h_recojet_matched_pt, h_recojet_pt, 1, 1, "B");
+        h_ratePUJet = (TH1F*)h_recojet_unmatched_pt->Clone();
+        h_ratePUJet->SetName("h_ratePUJet");
+        h_ratePUJet->Scale(1. / double(nEvents));


What if nEvents is 0?

@peteryouBuffalo please consider this comment from @mmusich
Would it be correct to just continue after L126 if nEvents = 0?

Unfortunately, I am not the original author of the changes that were made. However, I can definitely see there being an issue with the case if nEvents == 0. I will try to reach out and discuss with my colleagues at Buffalo who originally started and helped with the original PR (#39902), so we can figure out resolving this specific case. My assumption is that it should be fine to just continue if nEvents == 0, but I will check with others to confirm.

for the record, this was addressed in #43530

cmsbuild added this to the CMSSW_14_0_X milestone Nov 16, 2023

cmsbuild added dqm-pending operations-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 16, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 16, 2023

cmsbuild added the pf label Nov 18, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 18, 2023

cmsbuild added tests-approved and removed tests-started labels Nov 18, 2023

makortel reviewed Nov 20, 2023

View reviewed changes

Fixing issues in Garvita's PR with empty plots

0adf85b

peteryouBuffalo force-pushed the updatePFVal_13_3_X branch from 0715759 to 0adf85b Compare November 20, 2023 17:49

cmsbuild added tests-pending code-checks-pending and removed operations-pending tests-approved code-checks-approved labels Nov 20, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 20, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 21, 2023

cmsbuild added tests-approved and removed tests-started labels Nov 21, 2023

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Nov 25, 2023

rappoccio mentioned this pull request Nov 28, 2023

Adding jet purity and efficiency plots to PFValidation package #39902

Closed

cmsbuild added orp-approved and removed orp-pending labels Nov 28, 2023

cmsbuild merged commit 6b19bb3 into cms-sw:master Nov 28, 2023

perrotta mentioned this pull request Nov 30, 2023

Spurious comparison differences in workflows 136.X, 141.044, 4.54, ... #43456

Closed

mmusich reviewed Dec 2, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UPDATED: Adding of jet purity and efficiency plots to PFValidation package #43304

UPDATED: Adding of jet purity and efficiency plots to PFValidation package #43304

peteryouBuffalo commented Nov 16, 2023

cmsbuild commented Nov 16, 2023

cmsbuild commented Nov 16, 2023

swagata87 commented Nov 18, 2023

swagata87 commented Nov 18, 2023

cmsbuild commented Nov 18, 2023

makortel Nov 20, 2023

laurenhay Nov 20, 2023

makortel Nov 20, 2023

laurenhay Nov 20, 2023 •

edited

Loading

laurenhay Nov 20, 2023

cmsbuild commented Nov 20, 2023

cmsbuild commented Nov 20, 2023

tjavaid commented Nov 21, 2023

cmsbuild commented Nov 21, 2023

tjavaid commented Nov 25, 2023

cmsbuild commented Nov 25, 2023

rappoccio commented Nov 27, 2023

laurenhay commented Nov 27, 2023

rappoccio commented Nov 28, 2023

mmusich Dec 2, 2023

perrotta Dec 4, 2023

peteryouBuffalo Dec 4, 2023 •

edited

Loading

mmusich Dec 12, 2023 •

edited

Loading

UPDATED: Adding of jet purity and efficiency plots to PFValidation package #43304

UPDATED: Adding of jet purity and efficiency plots to PFValidation package #43304

Conversation

peteryouBuffalo commented Nov 16, 2023

PR description:

PR validation:

cmsbuild commented Nov 16, 2023

cmsbuild commented Nov 16, 2023

swagata87 commented Nov 18, 2023

swagata87 commented Nov 18, 2023

cmsbuild commented Nov 18, 2023

Comparison Summary

makortel Nov 20, 2023

Choose a reason for hiding this comment

laurenhay Nov 20, 2023

Choose a reason for hiding this comment

makortel Nov 20, 2023

Choose a reason for hiding this comment

laurenhay Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

laurenhay Nov 20, 2023

Choose a reason for hiding this comment

cmsbuild commented Nov 20, 2023

cmsbuild commented Nov 20, 2023

tjavaid commented Nov 21, 2023

cmsbuild commented Nov 21, 2023

Comparison Summary

tjavaid commented Nov 25, 2023

cmsbuild commented Nov 25, 2023

rappoccio commented Nov 27, 2023

laurenhay commented Nov 27, 2023

rappoccio commented Nov 28, 2023

mmusich Dec 2, 2023

Choose a reason for hiding this comment

perrotta Dec 4, 2023

Choose a reason for hiding this comment

peteryouBuffalo Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

mmusich Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

laurenhay Nov 20, 2023 •

edited

Loading

peteryouBuffalo Dec 4, 2023 •

edited

Loading

mmusich Dec 12, 2023 •

edited

Loading