-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pull Request to Add Local and Component-wise Prediction Rigidity #209
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add tests into the tests/test_metrics.py
? You can take the ReconstructionMeasuresTests
class in the file as reference
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks looks good, just had minor comments. Can u also add your code doc to the sphinx doc? You just need to add the prediction rigidity to this file https://github.com/scikit-learn-contrib/scikit-matter/blob/main/docs/src/references/metrics.rst using autofunction as for the reconstruction measures. The text of the automodule can be changed in this file https://github.com/scikit-learn-contrib/scikit-matter/blob/main/src/skmatter/metrics/__init__.py At the moment it only mentions the reconstructions measures. Feel free to change the text so it includes better both measures and ask questions, sphinx doc is a bit confusing and frustrating at the beinning.
* implemented the LPR and (L)CPR into the metrics module and wrote up the documentation and tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
Hello all,
This PR aims to introduce new features to
scikit-matter
, which allows the user to calculate what is called the local prediction rigidity (LPR) and the (local) component-wise prediction rigidity ((L)CPR).LPR and CPR are metrics that allow one to assess the robustness, or quite literally the "rigidity", in the local or component-wise prediction that the machine learning model makes. More details can be found in this preprint: https://arxiv.org/pdf/2306.15638.pdf
(Note that CPR is not explained in the above preprint. It is, however, a metric that follows the same spirit as the LPR, and is something that will be shared in forthcoming works by myself and coworkers.)
I found these things to best fit in with the
metrics
, and hence the functions have been coded in there. Key points to note, among other things, is that the function assumes the user to (1) provide the regularization strength of a model that has been trained, and (2) consider the LPR and CPR only in the contexts where training is done on structural average feature vectors and targets, to keep the resulting metric within an interpretable and roughly transferable range.Please let me know what you think. Tagging and assigning @agoscinski as he has helped me write up this PR. Will only post as draft at the moment.
This PR should close #191
📚 Documentation preview 📚: https://scikit-matter--209.org.readthedocs.build/en/209/