Move DD4Hep workflows from 13TeV to 14TeV #32857

silviodonato · 2021-02-09T17:51:22Z

This PR moves TTbar and ZMM DD4Hep workflows from 13 TeV to 14 TeV, as discussed today at the ORP meeting. The reason is that 11634.0 is used in the current RelVal production and then we want the corresponding 11634.911. Same thing for 11642.911 which has been moved to 14 TeV 11650.911. The short matrix has been updated accordingly.
The PR has been tested simply with runTheMatrix.py -l 11634.911,11650.911

silviodonato · 2021-02-09T17:51:53Z

please test 11634.911

cc: @cms-sw/simulation-l2 @cms-sw/geometry-l2

davidlange6 · 2021-02-09T17:55:11Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -950,7 +950,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
            dd4hepGeom+=stepDict[step][k]['--geometry']
            stepDict[stepName][k] = merge([{'--geometry' : dd4hepGeom, '--procModifiers': 'dd4hep'}, stepDict[step][k]])
    def condition(self, fragment, stepList, key, hasHarvest):
-        return ((fragment=='TTbar_13' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \
+        return ((fragment=='TTbar_13' or fragment=='TTbar_14' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \


since we won't use dd4hep for run2, perhaps 13 tev workflows aren't very useful?

Since we have only 11624.911 in pre2, I think it is good to have both 11624.911 and 11634.911 in pre3

I also agreed with @davidlange6 that 13TeV isn't useful to keep

cmsbuild · 2021-02-09T17:57:53Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32857/21079

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_2017.py modified in PR(s): Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765, Patatrack integration - Pixel workflows (12/N) #31854, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Track classifier using TensorFlow - continuation #32128
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765, Patatrack integration - Pixel workflows (12/N) #31854, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Track classifier using TensorFlow - continuation #32128, Phase2 Tracker description change of paradigm, to be compatible with DD4hep #32843

cmsbuild · 2021-02-09T17:58:15Z

A new Pull Request was created by @silviodonato (Silvio Donato) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @srimanob can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cvuosalo · 2021-02-09T21:57:06Z

@cmsbuild please test

cvuosalo · 2021-02-09T21:57:59Z

@silviodonato Thank you for creating this workflow.

cmsbuild · 2021-02-10T00:27:55Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1d68c5/12808/summary.html
COMMIT: 01b39f0
CMSSW: CMSSW_11_3_X_2021-02-09-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32857/12808/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2751765
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2751742
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

chayanit · 2021-02-10T03:56:23Z

Configuration/PyReleaseValidation/python/relval_2017.py

@@ -55,7 +55,7 @@
           10824.501,
           10824.511,
           10824.521,
-           11624.911, 11642.911,
+           11634.911,11624.911, 11642.911,


@silviodonato , @cvuosalo could you clarify why we need 13TeV for Run3 scenario? Shall we move everything about DD4HEP to 14TeV (also ZMM)?

chayanit · 2021-02-10T03:57:12Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -950,7 +950,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
            dd4hepGeom+=stepDict[step][k]['--geometry']
            stepDict[stepName][k] = merge([{'--geometry' : dd4hepGeom, '--procModifiers': 'dd4hep'}, stepDict[step][k]])
    def condition(self, fragment, stepList, key, hasHarvest):
-        return ((fragment=='TTbar_13' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \
+        return ((fragment=='TTbar_13' or fragment=='TTbar_14' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \


I also agreed with @davidlange6 that 13TeV isn't useful to keep

davidlange6 · 2021-02-10T08:18:52Z

For what purpose? Presumably the goal is to compare a dd4hep geometry to a dd geometry - the latter has 14 TeV samples for run3.

…

On Feb 9, 2021, at 7:03 PM, Silvio Donato ***@***.***> wrote: @silviodonato commented on this pull request. In Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py: > @@ -950,7 +950,7 @@ def setup_(self, step, stepName, stepDict, k, properties): dd4hepGeom+=stepDict[step][k]['--geometry'] stepDict[stepName][k] = merge([{'--geometry' : dd4hepGeom, '--procModifiers': 'dd4hep'}, stepDict[step][k]]) def condition(self, fragment, stepList, key, hasHarvest): - return ((fragment=='TTbar_13' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \ + return ((fragment=='TTbar_13' or fragment=='TTbar_14' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \ Since we have only 11624.911 in pre2, I think it is good to have both 11624.911 and 11634.911 in pre3 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

…to 14TeV

silviodonato · 2021-02-10T11:39:18Z

please test workflows 11634.911, 11650.911

silviodonato · 2021-02-10T11:41:34Z

As you prefer, I've removed the 13TeV workflow, and so I've also updated the short matrix to replace the DD4HEP worflow from 11624.911 to 11634.911. I also updated the ZMM to 14TeV, just to have everything at 14TeV (11642.911 -> 11650.911)

cmsbuild · 2021-02-10T11:46:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32857/21093

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_2017.py modified in PR(s): Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765, Patatrack integration - Pixel workflows (12/N) #31854, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Track classifier using TensorFlow - continuation #32128
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765, Patatrack integration - Pixel workflows (12/N) #31854, Phase2 Tracker description change of paradigm, to be compatible with DD4hep #32843, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Track classifier using TensorFlow - continuation #32128

cmsbuild · 2021-02-10T11:46:25Z

Pull request #32857 was updated. @chayanit, @srimanob, @wajidalikhan, @kpedro88, @jordan-martins can you please check and sign again.

cmsbuild · 2021-02-10T14:07:47Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1d68c5/12826/summary.html
COMMIT: 53711db
CMSSW: CMSSW_11_3_X_2021-02-09-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32857/12826/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1d68c5/11634.911_TTbar_14TeV+2021_DD4hep+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1d68c5/11650.911_ZMM_14+2021_DD4hep+ZMM_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2536530
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2536507
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 151 log files, 37 edm output root files, 36 DQM output files

cvuosalo · 2021-02-10T17:37:07Z

The logs from the new workflows look good. They seem to have run correctly.

silviodonato · 2021-02-11T08:43:56Z

I've updated the PR description. @cms-sw/upgrade-l2 @cms-sw/pdmv-l2 do you have further comments?

chayanit · 2021-02-11T08:51:17Z

+1

silviodonato · 2021-02-11T18:25:52Z

merge

cmsbuild added this to the CMSSW_11_3_X milestone Feb 9, 2021

cmsbuild added code-checks-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Feb 9, 2021

davidlange6 reviewed Feb 9, 2021

View reviewed changes

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 9, 2021

cmsbuild mentioned this pull request Feb 9, 2021

Phase2 Tracker description change of paradigm, to be compatible with DD4hep #32843

Merged

cmsbuild added tests-started and removed tests-pending labels Feb 9, 2021

cmsbuild added tests-approved and removed tests-started labels Feb 10, 2021

chayanit reviewed Feb 10, 2021

View reviewed changes

silviodonato added 2 commits February 10, 2021 12:25

add TTbar_14TeV DD4HEP workflow

6225c60

remove 11624.911, add 11634.911 in the short matrix, move ZMM DD4HEP …

53711db

…to 14TeV

silviodonato force-pushed the TTbar_14TeV_DD4HEP_workflow branch from 01b39f0 to 53711db Compare February 10, 2021 11:38

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved tests-approved labels Feb 10, 2021

cmsbuild added tests-started and removed tests-pending labels Feb 10, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 10, 2021

cmsbuild added tests-approved and removed tests-started labels Feb 10, 2021

cmsbuild mentioned this pull request Feb 10, 2021

PPS pixel topology #32868

Merged

cmsbuild mentioned this pull request Feb 10, 2021

Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765

Closed

silviodonato changed the title ~~Addition of DD4Hep TTbar_14TeV workflow~~ Move DD4Hep workflows from 13TeV to 14TeV Feb 11, 2021

cmsbuild added pdmv-approved and removed pdmv-pending labels Feb 11, 2021

cmsbuild added orp-approved and removed orp-pending labels Feb 11, 2021

cmsbuild merged commit a09e085 into cms-sw:master Feb 11, 2021

slava77 mentioned this pull request Feb 16, 2021

11634.911 from a short matrix is added to the reco comparison workflow map file cms-sw/cms-bot#1490

Merged

qliphy mentioned this pull request Feb 22, 2021

Workflow 11634.911 appearing broken in several PR tests #32963

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move DD4Hep workflows from 13TeV to 14TeV #32857

Move DD4Hep workflows from 13TeV to 14TeV #32857

silviodonato commented Feb 9, 2021 •

edited

Loading

silviodonato commented Feb 9, 2021

davidlange6 Feb 9, 2021

silviodonato Feb 9, 2021

chayanit Feb 10, 2021

cmsbuild commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

cvuosalo commented Feb 9, 2021

cvuosalo commented Feb 9, 2021

cmsbuild commented Feb 10, 2021

chayanit Feb 10, 2021

chayanit Feb 10, 2021

davidlange6 commented Feb 10, 2021 via email

silviodonato commented Feb 10, 2021

silviodonato commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

cvuosalo commented Feb 10, 2021

silviodonato commented Feb 11, 2021

chayanit commented Feb 11, 2021

silviodonato commented Feb 11, 2021

Move DD4Hep workflows from 13TeV to 14TeV #32857

Move DD4Hep workflows from 13TeV to 14TeV #32857

Conversation

silviodonato commented Feb 9, 2021 • edited Loading

silviodonato commented Feb 9, 2021

davidlange6 Feb 9, 2021

Choose a reason for hiding this comment

silviodonato Feb 9, 2021

Choose a reason for hiding this comment

chayanit Feb 10, 2021

Choose a reason for hiding this comment

cmsbuild commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

cvuosalo commented Feb 9, 2021

cvuosalo commented Feb 9, 2021

cmsbuild commented Feb 10, 2021

Comparison Summary

chayanit Feb 10, 2021

Choose a reason for hiding this comment

chayanit Feb 10, 2021

Choose a reason for hiding this comment

davidlange6 commented Feb 10, 2021 via email

silviodonato commented Feb 10, 2021

silviodonato commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

Comparison Summary

cvuosalo commented Feb 10, 2021

silviodonato commented Feb 11, 2021

chayanit commented Feb 11, 2021

silviodonato commented Feb 11, 2021

silviodonato commented Feb 9, 2021 •

edited

Loading