Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEAT] CRPS, MSSE and Energy Score metrics #134

Merged
merged 7 commits into from
Dec 16, 2022
Merged

[FEAT] CRPS, MSSE and Energy Score metrics #134

merged 7 commits into from
Dec 16, 2022

Conversation

kdgutier
Copy link
Collaborator

General improvements to evaluation utilities

  • Added scaled_crps, msse and energy_score metrics to evaluation.py
  • Added running tests for metrics in probabilistic_methods.ipynb
  • Added multivariate scoring and hierarchical accuracy references
  • General improvements to the evaluation.ipynb documentation
  • Changed default output samples.shape from [samples, N,H] to [N,H,samples], updated dependencies accordingly

PENDING

  • Validate metrics using core.HierarchicalReconciliation in TourismL experiment.
  • Update core.HierarchicalReconciliation to produce online samples or store memory through fit, predict/sample.

@kdgutier kdgutier requested a review from cchallu December 15, 2022 23:14
@kdgutier kdgutier self-assigned this Dec 15, 2022
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

cchallu
cchallu approved these changes Dec 16, 2022
View reviewed changes
hierarchicalforecast/evaluation.py Outdated Show resolved Hide resolved
hierarchicalforecast/evaluation.py Outdated Show resolved Hide resolved
@kdgutier kdgutier merged commit 82b53db into main Dec 16, 2022
@kdgutier kdgutier deleted the revisited_evaluation branch December 16, 2022 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cchallu cchallu cchallu approved these changes

Assignees

@kdgutier kdgutier

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kdgutier @cchallu