Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull Request to Add Local and Component-wise Prediction Rigidity #209

Merged
merged 2 commits into from
Aug 21, 2023

Conversation

SanggyuChong
Copy link
Contributor

@SanggyuChong SanggyuChong commented Aug 14, 2023

Hello all,

This PR aims to introduce new features to scikit-matter, which allows the user to calculate what is called the local prediction rigidity (LPR) and the (local) component-wise prediction rigidity ((L)CPR).

LPR and CPR are metrics that allow one to assess the robustness, or quite literally the "rigidity", in the local or component-wise prediction that the machine learning model makes. More details can be found in this preprint: https://arxiv.org/pdf/2306.15638.pdf
(Note that CPR is not explained in the above preprint. It is, however, a metric that follows the same spirit as the LPR, and is something that will be shared in forthcoming works by myself and coworkers.)

I found these things to best fit in with the metrics, and hence the functions have been coded in there. Key points to note, among other things, is that the function assumes the user to (1) provide the regularization strength of a model that has been trained, and (2) consider the LPR and CPR only in the contexts where training is done on structural average feature vectors and targets, to keep the resulting metric within an interpretable and roughly transferable range.

Please let me know what you think. Tagging and assigning @agoscinski as he has helped me write up this PR. Will only post as draft at the moment.

This PR should close #191


📚 Documentation preview 📚: https://scikit-matter--209.org.readthedocs.build/en/209/

Copy link
Collaborator

@agoscinski agoscinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add tests into the tests/test_metrics.py? You can take the ReconstructionMeasuresTests class in the file as reference

src/skmatter/preprocessing/_data.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/preprocessing/_data.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
src/skmatter/metrics/_prediction_rigidities.py Outdated Show resolved Hide resolved
@SanggyuChong SanggyuChong marked this pull request as ready for review August 18, 2023 14:42
Copy link
Collaborator

@agoscinski agoscinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks looks good, just had minor comments. Can u also add your code doc to the sphinx doc? You just need to add the prediction rigidity to this file https://github.com/scikit-learn-contrib/scikit-matter/blob/main/docs/src/references/metrics.rst using autofunction as for the reconstruction measures. The text of the automodule can be changed in this file https://github.com/scikit-learn-contrib/scikit-matter/blob/main/src/skmatter/metrics/__init__.py At the moment it only mentions the reconstructions measures. Feel free to change the text so it includes better both measures and ask questions, sphinx doc is a bit confusing and frustrating at the beinning.

tests/test_metrics.py Show resolved Hide resolved
tox.ini Show resolved Hide resolved
* implemented the LPR and (L)CPR into the metrics module and wrote up the
documentation and tests.
Copy link
Collaborator

@agoscinski agoscinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@agoscinski agoscinski merged commit f48337f into scikit-learn-contrib:main Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implementation of local prediction rigidity
2 participants