Including systematics variation samples

[Tomoya-Iizawa]

Hi,

I would like to include different generator samples as systematics sample, but could someone tell me how I can do that?
I think systematics can be implemented as following,

Systematics:
  - Name: "Signal_modeling"
    Up:
      SamplePath: "modeling_variation_up.root"
    Down:
      SamplePath: "modeling_variation_down.root"
    Samples: "Signal"

but for the generator difference, I would like to include different file as ttbar-syst-Sh2212.root, and would like to use one of the histogram (not as Up and Down) in that file.

Best Regards,
Tomoya

[alexander-held]

Hi Tomoya, an example for how to set this up is here:

cabinetry/utils/config_histograms.yml

Lines 31 to 37 in a0638e1

	- Name: "Modeling"
	Up:
	VariationPath: "Modeling_Up"
	Down:
	Symmetrize: True
	Samples: "Background"
	Type: "NormPlusShape"

If you want to read a histogram from a different file you could set it up like this for example:

General:
    InputPath: "{SamplePath}:{RegionPath}"

Samples:
  - Name: "ttbar"
    SamplePath: "ttbar-PP8.root"

Systematics:
  - Name: "ttbar_Sherpa"
    Up:
      SamplePath: "ttbar-syst-Sh2212.root"
    Down:
      Symmetrize: True
    Samples: "ttbar"
    Type: "NormPlusShape" 

This will build the ttbar sample using histograms from a file ttbar-PP8.root and the ttbar_Sherpa variation by switching to ttbar-syst-Sh2212.root. See also https://cabinetry.readthedocs.io/en/latest/core.html#inputs-to-cabinetry-ntuples-or-histograms for the relevant documentation.

[alexander-held]

What you see here is an implementation decision taken by cabinetry: histosys modifiers are always normalized, while normsys modifiers absorb any effects that change normalization per channel (region). The reason behind this is that it helps avoid unphysical model predictions (negative yields) and helps stabilize the fit. A bit more details and an example are shown in these slides https://indico.belle2.org/event/8470/contributions/55829/ on page 16. Since both the normsys and histosys modifier are correlated by having the same name, you automatically still have the correct predictions at +/- 1 sigma. What changes is the exponential extrapolation of the histosys vs the linear extrapolation of the normsys, which starts mattering when the normalization effects are very large. There is no right or wrong here, it is a choice which one to pick. The sildes have an example and link to a gist you can use to try out other examples.

I just rebased the branch feat/custom-modifier-models (from #385) so it now includes all changes from v0.6.0. You should be able to use it now. I would like to still wait with merging that branch until we get an official pyhf release with these changes.

[Tomoya-Iizawa]

Thank you for the explanation and quick updates again!
OK, if this is working as expected, it is good.

About the development version, it still does not work with the following error:

Traceback (most recent call last):
File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/fitting/run_cabinetry.py", line 883, in
run_cabinetry(args["inputs"],
File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/fitting/run_cabinetry.py", line 168, in run_cabinetry
model, data = cabinetry.model_utils.model_and_data(
TypeError: model_and_data() got an unexpected keyword argument 'validate'

Could you check this?

[alexander-held]

I assume something went wrong with your installation of the version from the branch. The validate keyword works fine for me from that branch.

[Tomoya-Iizawa]

thanks, hm, ok, I may forget something to set up. I will try some more things.

[Tomoya-Iizawa]

After installation again, development version seems working!
Thank you again for your help, and I will try to check limit, yields, make plots and so on, and I may ask question when I have.