Shape parameters #58

hammannr · 2023-07-25T12:39:04Z

This adds shape parameter support for the blueice extended model (closes #33). Also, some bugs are fixed.

Some of the things I modified:

Make the data a dict (instead of a list) where the keys are the names of the likelihood term. For toy data, the generate_values are the last entry of the data but for real data this doesn't make sense. So to avoid ambiguities, I think it is better to clearly relate the data with a term of the likelihood (or the generate_values).
The ll.rate_parameters are missing the _rate_multiplier suffix so I updated the parameter overriding and filtering accordingly.
Add shifted ER templates (naively shifted up and down 1 and 2 bins) to illustrate shape parameters.
Update the minuit wrapper (now compatible with v>= 2.21)
The errordef was so far not correctly assigned, which caused trouble with the correct error estimation. I also made the minuit object a member of the class, so we can do statistical_model.minuit_object.minos() after the fit to very efficiently compute the (asymmetric) uncertainties of all fit parameters!
Fixed some indentation bug in the search for the template folder
Enabled str-type parameter space definitions as before.
Some adjustments to legacy code just to do some cross-checks and debugging.

You can try it using the following code:

from alea.blueice_extended_model import BlueiceExtendedModel

config_path = "alea/examples/model_configs/unbinned_wimp_statistical_model.yaml"

statistical_model = BlueiceExtendedModel.from_config(config_path)
statistical_model.data = statistical_model.generate_data()
best_fit_res, best_fit_nll = statistical_model.fit(verbose=True)
# best fit of er_band_shift should be around 0

statistical_model = BlueiceExtendedModel.from_config(config_path)
statistical_model.data = statistical_model.generate_data(er_band_shift=1.2)
best_fit_res, best_fit_nll = statistical_model.fit(verbose=True)
# best fit of er_band_shift should be around 1.2

…_parameters

github-actions · 2023-07-25T12:40:19Z

Pull Request Test Coverage Report for Build 5673114176

50 of 62 (80.65%) changed or added relevant lines in 6 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.4%) to 17.915%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
alea/_likelihoods/ll_nt_from_config.py	0	3	0.0%
alea/template_source.py	0	4	0.0%
alea/utils.py	9	14	64.29%

Files with Coverage Reduction	New Missed Lines	%
alea/parameters.py	1	81.25%
alea/utils.py	2	79.41%

Totals
Change from base Build 5671363159:	0.4%
Covered Lines:	524
Relevant Lines:	2925

💛 - Coveralls

alea/model.py

alea/template_source.py

alea/models/blueice_extended_model.py

hammannr · 2023-07-25T13:25:16Z

I now updated the exmple to look for the config in alea/model_configs but I'm still in favor of re-introducing the alea/examples folder for configs and templates.

Edit: I re-introduced the folder now as I think there should be no incentive for anyone to add their templates and configs in alea directly. If there are strong opinions against that we can discuss and revert this 😊

github-actions · 2023-07-25T13:44:46Z

alea/models/__init__.py

@@ -1,5 +1,5 @@
-from . import gaussian_model
-from .gaussian_model import *
+from ..examples import gaussian_model


[pep8] _{reported by reviewdog 🐶}
D104 Missing docstring in public package

alea/models/__init__.py

kdund

At the moment, the changes makes it so that fits change if all parameters are set:

from alea.examples import gaussian_model
simple_model = gaussian_model.GaussianModel()
simple_model.data = simple_model.generate_data(mu=0, sigma=2)
bf, ret = simple_model.fit(sigma=2, mu=2)
bf, ret

Fails with

----> 1 m = Minuit(MinuitWrap(cost, simple_model.parameters),mu=0, sigma=0)

File ~/opt/miniconda3/envs/myenv/lib/python3.8/site-packages/iminuit/minuit.py:625, in Minuit.__init__(***failed resolving arguments***)
    617 self._strategy = MnStrategy(1)
    618 self._fcn = FCN(
    619     fcn,
    620     getattr(fcn, "grad", grad),
    621     array_call,
    622     getattr(fcn, "errordef", 1.0),
    623 )
--> 625 self._init_state = _make_init_state(self._pos2var, start, kwds)
    626 self._values = mutil.ValueView(self)
    627 self._errors = mutil.ErrorView(self)

File ~/opt/miniconda3/envs/myenv/lib/python3.8/site-packages/iminuit/minuit.py:2470, in _make_init_state(pos2var, args, kwds)
   2468     for kw in kwds:
   2469         if kw not in pos2var:
-> 2470             raise RuntimeError(
   2471                 f"{kw} is not one of the parameters [{' '.join(pos2var)}]"
   2472             )
   2473     nargs = len(kwds)
   2475 if len(pos2var) != nargs:

RuntimeError: mu is not one of the parameters []

We'll have to short-circuit this possibility somehow

alea/model.py

hammannr · 2023-07-25T15:52:30Z

Ah @kdund I think this is an iminuit version problem. I developed with an environment, which contained python 3.9 and iminuit 2.22 and then I got a lot of warnings so I updated the implementation. Now however it only works with iminuit >= 2.21, I also updated the requirements.txt accordingly.

alea/model.py

kdund · 2023-07-25T18:12:31Z

alea/utils.py

@@ -10,15 +10,18 @@ def get_analysis_space(analysis_space: dict) -> list:

    for element in analysis_space:
        for key, value in element.items():
-            if value.startswith("np."):
+            if isinstance(value, str) and value.startswith("np."):


Do you have an example for usage of this?

The example config file has the analysis space defined in the following way:

analysis_space: - "cs1": 'np.arange(0, 102, 2)' - "cs2": 'np.geomspace(100, 100000, 51)'

This will be evaluated accordingly and saves a lot of space compared to the implementation in binference. However, I also re-enabled the binference-like definition, which can be useful when dealing with irregular binnings. To test this you could replace the definition of cs1 with the following:

- "cs1": "0. 2. 4. 6. 8. 10. 12. 14. 16. 18. 20. 22. 24. 26. 28. 30. 32. 34. 36. 38. 40. 42. 44. 46. 48. 50. 52. 54. 56. 58. 60. 62. 64. 66. 68. 70. 72. 74. 76. 78. 80. 82. 84. 86. 88. 90. 92. 94. 96. 98. 100."

and it should do exactly the same thing.

Also, the example I posted in slack features both implementations (irregular binning in R).

alea/models/blueice_extended_model.py

alea/utils.py

requirements.txt

alea/template_source.py

alea/models/blueice_extended_model.py

hammannr · 2023-07-26T18:05:32Z

@dachengx is now reviewdog 2.0 😄 But doesn't harm to be reminded to write readable code. I'll make those changes once the rest is fixed and the tests are running.

alea/model_configs/unbinned_wimp_statistical_model.yaml

alea/utils.py

tests/test_blueice_extended_model.py

alea/utils.py

alea/models/blueice_extended_model.py

hammannr · 2023-07-27T08:12:13Z

Thanks a lot, @dachengx for fixing the wildcard issue -- very nice and clean solution! 😊 I now also implemented some of the formatting suggestions. Do you have any other suggestions regarding the code and have you tried running it?

Also, @kdund did changing the iminuit version solve the issue and are you happy with the implementation of the analysis_space? 😊

dachengx

Thanks @hammannr . Looks good to me. I agree that we put the code-style cleaning in another PR.

dachengx · 2023-07-27T12:27:27Z

alea/utils.py

@@ -140,6 +141,8 @@ def get_template_folder_list(likelihood_config):
        template_folder_list = [likelihood_config["template_folder"]]
    elif isinstance(likelihood_config["template_folder"], list):
        template_folder_list = likelihood_config["template_folder"]
+    elif likelihood_config["template_folder"] is None:


Great that you find this case!

kdund

The fitting works for me w/ updated iminuit, thanks, @hammannr!
Slicing and stuff we can look at later.

coveralls · 2024-12-02T12:37:56Z

Pull Request Test Coverage Report for Build 5666611708

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 33 (0.0%) changed or added relevant lines in 6 files are covered.
142 unchanged lines in 5 files lost coverage.
Overall coverage remained the same at 0.0%

Changes Missing Coverage	Changed/Added Lines	%
alea/simulators.py	1	0.0%
alea/model.py	2	0.0%
alea/_likelihoods/ll_nt_from_config.py	3	0.0%
alea/template_source.py	3	0.0%
alea/utils.py	6	0.0%
alea/models/blueice_extended_model.py	18	0.0%

Files with Coverage Reduction	New Missed Lines	%
alea/template_source.py	1	0.0%
alea/init.py	2	0.0%
alea/utils.py	13	0.0%
alea/models/blueice_extended_model.py	50	0.0%
alea/model.py	76	0.0%

Totals
Change from base Build 5655453593:	0.0%
Covered Lines:	0
Relevant Lines:	2875

💛 - Coveralls

hammannr added 19 commits July 17, 2023 16:20

first implementation of shape parameters

cfd15c3

add shifted er templates

73b47c5

fix shape parameter init indent

51e33cc

small fixes that were ovrelooked

ec68e8f

switch to dict formatted data

fd82c52

fix kwarg filetering

cfed681

fix nominal rate parameter treatment

2215cc1

Trying to befriend the dog

c0a28a7

iminuit requirement

d4257d4

first implementation of shape parameters

4495021

add shifted er templates

b850f78

fix shape parameter init indent

1412c69

small fixes that were ovrelooked

865c739

switch to dict formatted data

281ec07

fix kwarg filetering

def9195

fix nominal rate parameter treatment

8b3a457

Trying to befriend the dog

a56bdf9

iminuit requirement

d33cfa2

Merge branch 'shape_parameters' of github.com:XENONnT/alea into shape…

a2ee6b3

…_parameters

hammannr added enhancement New feature or request inference Statistics code labels Jul 25, 2023

hammannr requested review from kdund and dachengx July 25, 2023 12:39

github-actions bot reviewed Jul 25, 2023

View reviewed changes

comply with structured-array data format

fa0e94f

hammannr added 2 commits July 25, 2023 15:39

re-introduce examples folder

7a05c1e

fix init and config

889b276

github-actions bot reviewed Jul 25, 2023

View reviewed changes

kdund requested changes Jul 25, 2023

View reviewed changes

alea/model.py Show resolved Hide resolved

alea/model.py Outdated Show resolved Hide resolved

alea/model.py Outdated Show resolved Hide resolved

Move MinuitWrap outside

b44f678

github-actions bot reviewed Jul 25, 2023

View reviewed changes

alea/model.py Show resolved Hide resolved

alea/model.py Show resolved Hide resolved

kdund reviewed Jul 25, 2023

View reviewed changes

hammannr and others added 3 commits July 26, 2023 10:43

undo changing folders

b07019b

forgot the templates..

e337b09

Merge branch 'main' into shape_parameters

806235d

github-actions bot reviewed Jul 26, 2023

View reviewed changes

alea/models/blueice_extended_model.py Outdated Show resolved Hide resolved

alea/models/blueice_extended_model.py Outdated Show resolved Hide resolved

hammannr added 2 commits July 26, 2023 12:02

patch for files with placeholders

5a225dc

update folder

962e258

dachengx requested changes Jul 26, 2023

View reviewed changes

dachengx reviewed Jul 26, 2023

View reviewed changes

alea/model_configs/unbinned_wimp_statistical_model.yaml Outdated Show resolved Hide resolved

Find formatted filepath in get_file_path

ba4096a

dachengx reviewed Jul 26, 2023

View reviewed changes

alea/utils.py Show resolved Hide resolved

github-actions bot reviewed Jul 26, 2023

View reviewed changes

Happier code style

ab7dbdd

github-actions bot reviewed Jul 26, 2023

View reviewed changes

alea/utils.py Outdated Show resolved Hide resolved

alea/utils.py Outdated Show resolved Hide resolved

dachengx and others added 2 commits July 27, 2023 03:54

Happier code style

abb842b

Some formatting and small things

d074e6d

github-actions bot reviewed Jul 27, 2023

View reviewed changes

alea/models/blueice_extended_model.py Show resolved Hide resolved

alea/models/blueice_extended_model.py Show resolved Hide resolved

hammannr requested review from dachengx and kdund July 27, 2023 08:12

dachengx approved these changes Jul 27, 2023

View reviewed changes

kdund approved these changes Jul 27, 2023

View reviewed changes

hammannr merged commit bf0a0ca into main Jul 27, 2023

hammannr deleted the shape_parameters branch July 27, 2023 13:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shape parameters #58

Shape parameters #58

hammannr commented Jul 25, 2023 •

edited

Loading

github-actions bot commented Jul 25, 2023 •

edited

Loading

hammannr commented Jul 25, 2023 •

edited

Loading

github-actions bot Jul 25, 2023

kdund left a comment

hammannr commented Jul 25, 2023 •

edited

Loading

kdund Jul 25, 2023

hammannr Jul 26, 2023

hammannr Jul 26, 2023

hammannr commented Jul 26, 2023

hammannr commented Jul 27, 2023

dachengx left a comment

dachengx Jul 27, 2023

kdund left a comment

coveralls commented Dec 2, 2024 •

edited

Loading

Shape parameters #58

Shape parameters #58

Conversation

hammannr commented Jul 25, 2023 • edited Loading

github-actions bot commented Jul 25, 2023 • edited Loading

Pull Request Test Coverage Report for Build 5673114176

💛 - Coveralls

hammannr commented Jul 25, 2023 • edited Loading

github-actions bot Jul 25, 2023

Choose a reason for hiding this comment

kdund left a comment

Choose a reason for hiding this comment

hammannr commented Jul 25, 2023 • edited Loading

kdund Jul 25, 2023

Choose a reason for hiding this comment

hammannr Jul 26, 2023

Choose a reason for hiding this comment

hammannr Jul 26, 2023

Choose a reason for hiding this comment

hammannr commented Jul 26, 2023

hammannr commented Jul 27, 2023

dachengx left a comment

Choose a reason for hiding this comment

dachengx Jul 27, 2023

Choose a reason for hiding this comment

kdund left a comment

Choose a reason for hiding this comment

coveralls commented Dec 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 5666611708

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

hammannr commented Jul 25, 2023 •

edited

Loading

github-actions bot commented Jul 25, 2023 •

edited

Loading

hammannr commented Jul 25, 2023 •

edited

Loading

hammannr commented Jul 25, 2023 •

edited

Loading

coveralls commented Dec 2, 2024 •

edited

Loading