Process dump python psets first [or how to modify outputCommands from ConfigBuilder]

[slava77]

Context: DataProcessing-like configuration building is done with an instance of cms.Process and then manipulations by ConfigBuilder, the final configuration is made by Process::dumpPython call (there is no intermediate text config file like for configurations made by cmsDriver).

Symptoms: process instance made by ConfigBuilder in python can not have its output module outputCommands modified. (Note that customization works in python text config files.)
More explicitly, say, for the event content defined by RECOEventContent, it is possible to modify the RECOEventContent PSet by just calling process.someOuput.RECOEventContent.remove(something), but it will have no effect on the output modules creaded by ConfigBuilder.
By default the python output module objects made by ConfigBuilder have the content of outputCommands inlined immediately, which looses the relationship with the original PSet. This behavior can be changed by inlineEventContent = False option:
it overrides the dict["dumpPython"] for outputCommands and leaves e.g. process.someOutput.outputCommands = RECOEventContent.outputCommands unchanged (not inlined). After this modification the process.dumpPython will leave process.someOutput.outputCommands = RECOEventContent.outputCommands as is and the dumped python is not runnable.

Proposed solution: change the order of dumps in Process::dumpPython
https://github.com/cms-sw/cmssw/blob/CMSSW_7_6_3/FWCore/ParameterSet/python/Config.py#L708
Put the self.psets and self.vpsets (and maybe more) first, so that they can be interpreted inside the modules using the psets.

[cmsbuild]

A new Issue was created by @slava77 (Slava Krutelyov).

@davidlange6, @smuzaffar, @Degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

Following commands in first line of a comment are recognized

• +1|approve[d]|sign[ed]: L1/L2's to approve it
• -1|reject[ed]: L1/L2's to reject it
• assign <category>[,<category>[,...]]: L1/L2's to request signatures from other categories
• unassign <category>[,<category>[,...]]: L1/L2's to remove signatures from other categories
• hold: L1/all L2's/release manager to mark it as on hold
• unhold: L1/user who put this PR on hold
• merge: L1/release managers to merge this request

[slava77]

looks like I can't assign issues to categories (at least not to core)

[Dr15Jones]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

@slava77 can you provide a simple example of a configuration that when one does process.dumpPython() the resultant output is unrunnable?

[slava77]

in CMSSW_7_5_6 (just because I had a test setup there)

• pick slava77:CMSSW_7_5_6/sign636/dataProcessingOutputs and compile
• run the following to produce what comes out from Process::dumpPython call somewhere inside the DataProcessing setup
  python Configuration/DataProcessing/test/RunPromptReco.py --scenario ppRun2 --global-tag 75X_dataRun2_v3 --lfn /whatever --reco --dqmio --alcareco TkAlMuonIsolated
• cmsRun RunPromptRecoCfg.py : as is it will produce an error

python encountered the error: <type 'exceptions.AttributeError'>
'Process' object has no attribute 'ALCARECOEventContent'

• you can edit RunPromptRecoCfg.py manually by moving the *EventContent PSets from the end to the beginning or by picking the change from slava77:CMSSW_7_5_6/sign636/dataProcessingOutputs-wPythonDump CMSSW_7_5_6...slava77:CMSSW_7_5_6/sign636/dataProcessingOutputs-wPythonDump (recompilation in this case will take a while)

Hope this makes it easier to follow.

[Dr15Jones]

Maybe posting the part of the configuration which is trying to access ALCARECOEventContent would be sufficient.

[slava77]

I copied the config to ~slava77/public/py/RunPromptRecoCfg_issue12842.py on lxplus

[Dr15Jones]

It took a while but I at least tracked down exactly what was happening. In Configuration/Applications/python/ConfigBuilder.py we have the following code

        if not self._options.inlineEventContent and hasattr(self.process,theStreamType+"EventContent"):
                def doNotInlineEventContent(instance,label = "cms.untracked.vstring(process."+theStreamType+"EventContent.outputCommands)"):
                    return label
                outputModule.outputCommands.__dict__["dumpPython"] = doNotInlineEventContent

What this code does is remove the Core approved standard member fuction dumpPython for the instance of the class held by outputModule.outputCommands and replaces it with the function doNotInlineEventContent. As far as I can tell there are no tests that such behavior should ever work.

@davidlange6 Is this really a behavior you want us to support?

[Dr15Jones]

At the Core meeting we decided to have Process.dumpPython write all known PSets first. This allows the code to work but does not address what I consider a very poor behaviour inside ConfigBuilder.py.

[Dr15Jones]

#12870 makes the requested change

[Dr15Jones]

+1

[cmsbuild]

This issue is fully signed and ready to be closed.