Merge to main for version 2.0.0

.gitignore

@@ -91,6 +91,7 @@ ENV/
 env.bak/
 venv.bak/
 .vscode
+.idea
 
 # Rope project settings
 .ropeproject

.pre-commit-config.yaml

@@ -1,7 +1,7 @@
 ---
 repos:
   - repo: https://github.com/psf/black
-    rev: 23.3.0
+    rev: 24.1.0
     hooks:
       - id: black
         name: black
@@ -26,7 +26,7 @@ repos:
       - id: isort
         entry: isort --profile=black
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.3.0
+    rev: v1.13.0
     hooks:
       - id: mypy
   - repo: local

.zenodo.json

@@ -16,7 +16,7 @@
         },
         {
             "orcid": "https://orcid.org/0009-0009-3207-4876",
-            "affiliation": "Bureau of Meteorology, Australia",
+            "affiliation": "Work undertaken while at the Bureau of Meteorology, Australia",
             "name": "Cook, Harrison"
         },
         {
@@ -56,7 +56,25 @@
             "orcid": "https://orcid.org/0000-0002-5545-1736",
             "affiliation": "CSIRO, Australia",
             "name": "Shrestha, Durga"
-        }                     
+        },
+        {
+            "orcid": "https://orcid.org/0009-0002-8569-1439",
+            "affiliation": "Independent Contributor, Australia",
+            "name": "Bishop, Sam"
+        },
+        {
+            "affiliation": "Independent Contributor, Australia",
+            "name": "Fisher, A.J."
+        },
+        {
+            "affiliation": "Independent Contributor, Australia",
+            "name": "Jinghan, Fu"
+        },
+        {
+            "orcid": "https://orcid.org/0009-0006-7361-163X",
+            "affiliation": "Independent Contributor, Australia",
+            "name": "Bluett, Liam"
+        }
     ],    
     "license": "Apache-2.0",
 

README.md

@@ -20,11 +20,11 @@ Below is a **curated selection** of the metrics, tools and statistical tests inc
 |-----------------------	|-----------------	|--------------	|
 | **[Continuous](https://scores.readthedocs.io/en/stable/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|MAE, MSE, RMSE, Additive Bias, Multiplicative Bias, Percent Bias, Pearson's Correlation Coefficient, Kling-Gupta Efficiency, Flip-Flop Index, Quantile Loss, Quantile Interval Score, Interval Score, Murphy Score, and threshold weighted scores for expectiles, quantiles and Huber Loss.             	|
 | **[Probability](https://scores.readthedocs.io/en/stable/included.html#probability)**        |Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                   |Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Functions (CDF) and ensembles (including threshold weighted versions), Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams).               |
-| **[Categorical](https://scores.readthedocs.io/en/stable/included.html#categorical)**       	|Scores for evaluating forecasts of categories.                	|17 binary contingency table (confusion matrix) metrics and the FIxed Risk Multicategorical (FIRM) Score.               	|
+| **[Categorical](https://scores.readthedocs.io/en/stable/included.html#categorical)**       	|Scores for evaluating forecasts of categories.                	|18 binary contingency table (confusion matrix) metrics and the FIxed Risk Multicategorical (FIRM) Score.               	|
 | **[Spatial](https://scores.readthedocs.io/en/stable/included.html#spatial)** 	|Scores that take into account spatial structure.                 	|Fractions Skill Score.              	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/stable/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
 | **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Cumulative Density Function Manipulation.              	|
-
+| **[Emerging](https://scores.readthedocs.io/en/stable/included.html#emerging)**        	|Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process.                 	|Risk Matrix Score.             	|
 
 `scores` not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.
 
@@ -74,10 +74,16 @@ array(2.)
 ```
 [Jupyter Notebook tutorials](https://scores.readthedocs.io/en/stable/tutorials/Tutorial_Gallery.html) are provided for each metric and statistical test in `scores`, as well as for some of the key features of `scores` (e.g., [dimension handling](https://scores.readthedocs.io/en/stable/tutorials/Dimension_Handling.html) and [weighting results](https://scores.readthedocs.io/en/stable/tutorials/Weighting_Results.html)). 
 
+To watch a PyCon AU 2024 conference presentation about `scores` [click here](https://youtu.be/jyq2jOqtXe0?si=HYoW1cNiplbb3R0c).
+
 ## Finding, Downloading and Working With Data
 
 All metrics, statistical techniques and data processing tools in `scores` work with [xarray](https://xarray.dev). [Some metrics](https://scores.readthedocs.io/en/stable/included.html#pandas) work with [pandas](https://pandas.pydata.org/). As such, `scores` works with any data source for which xarray or pandas can be used. See the [data sources](https://scores.readthedocs.io/en/stable/data.html) page and [this tutorial](https://scores.readthedocs.io/en/stable/tutorials/First_Data_Fetching.html) for more information on finding, downloading and working with different sources of data.
 
+## Archives of `scores` on Zenodo
+
+`scores` is archived on Zenodo. [Click here](https://doi.org/10.5281/zenodo.12697241) to see the latest version on Zenodo.
+
 ## Acknowledging or Citing `scores`
 
 If you use `scores` for a published work, we would appreciate you citing our [paper](https://doi.org/10.21105/joss.06889):

docs/api.md

@@ -51,6 +51,7 @@
 .. autofunction:: scores.probability.murphy_thetas
 .. autofunction:: scores.probability.roc_curve_data
 .. autofunction:: scores.probability.brier_score
+.. autofunction:: scores.probability.brier_score_for_ensemble
 .. autofunction:: scores.probability.isotonic_fit
 ```
 
@@ -109,3 +110,9 @@
 .. autofunction:: scores.pandas.continuous.mae
 ```
 
+## scores.emerging
+```{eval-rst}
+.. autofunction:: scores.emerging.risk_matrix_score
+.. autofunction:: scores.emerging.matrix_weights_to_array
+.. autofunction:: scores.emerging.weights_from_warning_scaling
+```

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "1.3.0"
+release = "2.0.0"
 
 version = __version__
 

docs/contributing.md

@@ -25,15 +25,23 @@ Please submit bug reports, feature requests and feedback as issues in GitHub: [h
 
 Please see the information provided in [SECURITY.md](SECURITY.md)
 
+## Creating Your Own Fork  of `scores` for the First Time
+
+Unless you are an advanced Git user, we would recommend you follow this process:
+
+1. First (i.e. **before** cloning the `scores` repository) create your own fork using the GitHub web user interface.
+2. Clone **your fork**. (Do not clone github.com/nci/scores).
+3. Immediately create a new local branch, with a command such as `git checkout -b branch_name`.
+
 ## Workflow for Submitting Pull Requests
 
 Prior to developing a pull request, it may be a good idea to create a [GitHub issue](https://github.com/nci/scores/issues) to capture what the pull request is trying to achieve, any pertinent details, and (if applicable) how it aligns to the roadmap. Otherwise, please explain this in the pull request.
 
 To submit a pull request, please use the following workflow: 
-1. create a fork of the `scores` repository, 
-2. create a feature branch on your fork,
-3. keep your feature branch rebased and up-to-date with the `scores` develop branch,
-4. when ready, submit a pull request to the develop branch of `scores`.
+
+1. Ensure you are working on a new feature branch in **your fork** (see [the section above](#creating-your-own-fork-of-scores-for-the-first-time)).
+2. Keep your feature branch rebased and up-to-date with the develop branch of `scores`. You can do this by first syncing the develop branch on your fork, and then rebase your feature branch against the develop branch on your fork.
+3. When ready, submit a pull request to the develop branch of github.com/nci/scores.
 
 To help disambiguate branches, some contributors like to prefix their branch names with a short numerical indentifier. This is up to the contributor and any approach to branch naming is welcome. 
 
@@ -110,6 +118,18 @@ To automate linter and fixer checks this project uses `pre-commit` which is set
 pre-commit install -t pre-commit -t pre-push
 ```
 
+## Pull Request Etiquette
+
+In general, the originator of a pull request will be the person who does all the coding work, including responding to feedback from others. Typically, feedback will be provided in the form of comments or code suggestions made through the GitHub web user interface.
+
+Sometimes, it may be pragmatic for the package maintainers to make or request a more complex code change. This typically occurs when a complex Git operation is needed to keep a pull request (PR) up to date, to resolve conflicts, or to make changes where the originator of the PR does not know how to proceed. It can also occur in the final stages of a PR lifecycle if there are small tidyups needed and time is a factor.
+
+Not every possibility can be accounted for, and the package maintainers will (if needed) push code directly to a PR to get something over the line. However, special circumstances aside, the maintainers will first leave a comment on the PR asking how the originator would like to proceed, so that there are no surprises.
+
+Most of these kinds of code change can also be handled as a PR by the reviewer onto the fork of the originator. This is slightly slower (i.e. does not take effect immediately), but allows for more control by the originator. This is probably most developer's general preference.
+
+In short - once you have made a PR, the maintainers may then take it, modify it, or include it as-is. However, every effort will be made to communicate about that process and make sure that the originator of the PR is happy with any modifications made.
+
 ## Review Processes
 
 All pull requests will undergo a review process prior to being included in the develop branch in order to ensure both the coding and the scientific validity of changes. 

docs/included.md

@@ -47,7 +47,7 @@
     [Griffiths et al. (2019)](https://doi.org/10.1002/met.1732); [Griffiths et al. (2021)](https://doi.org/10.1071/ES21010)
 * - Interval Score
   - [API](api.md#scores.continuous.interval_score)
-  - [Tutorial](project:./tutorials/Quantile_Interval_And_Interval_Scores.md)
+  - [Tutorial](project:./tutorials/Quantile_Interval_And_Interval_Score.md)
   - [Gneiting and Raftery (2007) - Section 6.2](https://doi.org/10.1198/016214506000001437)
 * - Isotonic Fit, *see Isotonic Regression*
   - —
@@ -115,7 +115,7 @@
   - —
 * - Quantile Interval Score
   - [API](api.md#scores.continuous.quantile_interval_score)
-  - [Tutorial](project:./tutorials/Quantile_Interval_And_Interval_Scores.md)
+  - [Tutorial](project:./tutorials/Quantile_Interval_And_Interval_Score.md)
   - [Winkler (1972) ](https://doi.org/10.2307/2284720)
 * - Quantile Loss (Quantile Score, Pinball Loss)
   - [API](api.md#scores.continuous.quantile_score)
@@ -168,6 +168,10 @@
   - [API](api.md#scores.probability.brier_score)
   - [Tutorial](project:./tutorials/Brier_Score.md)
   - [Brier (1950)](https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2)
+* - Brier Score for Ensembles
+  - [API](api.md#scores.probability.brier_score_for_ensemble)
+  - [Tutorial](project:./tutorials/Brier_Score.md)
+  - [Ferro (2013)](https://doi.org/10.1002/qj.2270)
 * - Continuous Ranked Probability Score (CRPS) for Cumulative Distribution Functions (CDFs)
   -    
   - 
@@ -399,6 +403,14 @@
     [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
   - 
     [Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#POD)
+* -  
+    - Negative Predictive Value
+  -
+    [API](api.md#scores.categorical.BasicContingencyManager.negative_predictive_value)
+  - 
+    [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
+  - 
+    [Wikipedia](https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values); [Monaghan et al. (2021)](https://doi.org/10.3390/medicina57050503)
 * -  
     - Odds Ratio
   -
@@ -424,7 +436,15 @@
   - 
     [Peirce (1884)](https://doi.org/10.1126/science.ns-4.93.453.b); [Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#HK)
 * -  
-    - Precision (Success Ratio)
+    - Positive Predictive Value (Success Ratio, Precision)
+  -
+    [API](api.md#scores.categorical.BasicContingencyManager.positive_predictive_value)
+  - 
+    [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
+  - 
+    [Wikipedia](https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values); [Success ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#SR); [Monaghan et al. (2021)](https://doi.org/10.3390/medicina57050503)
+* -  
+    - Precision (Success Ratio, Positive Predictive Value)
   -
     [API](api.md#scores.categorical.BasicContingencyManager.precision)
   - 
@@ -462,17 +482,17 @@
   - 
     [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
   - 
-    [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)
+    [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity); [Monaghan et al. (2021)](https://doi.org/10.3390/medicina57050503)
 * -  
     - Specificity (True Negative Rate)
   -
     [API](api.md#scores.categorical.BasicContingencyManager.specificity)
   - 
     [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
   - 
-    [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)
+    [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity); [Monaghan et al. (2021)](https://doi.org/10.3390/medicina57050503)
 * -  
-    - Success Ratio (Precision)
+    - Success Ratio (Precision, Positive Predictive Value)
   -
     [API](api.md#scores.categorical.BasicContingencyManager.success_ratio)
   - 
@@ -723,4 +743,36 @@
   - [Wikipedia](https://en.wikipedia.org/wiki/Root-mean-square_deviation)
 ```
 
+## Emerging
+
+```{Caution} 
+  This section of the API contains implementations of novel metrics that are still undergoing mathematical peer review. These implementations may change in line with the peer review process.
+```
 
+```{list-table}
+:header-rows: 1
+
+* - Name (Alphabetical order)
+  - API
+  - Tutorial
+  - Reference(s)
+* - Risk Matrix Score
+  - 
+  - 
+  - 
+* -  
+    - Risk Matrix Score
+  - [API](api.md#scores.emerging.risk_matrix_score)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+* -  
+    - Risk Matrix Score - Matrix Weights to Array
+  - [API](api.md#scores.emerging.matrix_weights_to_array)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+* -  
+    - Risk Matrix Score - Warning Scaling to Weight Array
+  - [API](api.md#scores.emerging.weights_from_warning_scaling)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+```

docs/related_works.md

@@ -2,6 +2,11 @@
 
 Here are some related packages that may be of interest.
 
+- **`Scoringrules`**
+	- Open source Python package that provides verification metrics with backends for [`NumPy`](https://github.com/numpy/numpy), [`JAX`](https://github.com/jax-ml/jax), [`PyTorch`](https://github.com/pytorch/pytorch) and [`TensorFlow`](https://github.com/tensorflow/tensorflow).
+	- [Software Repository](https://github.com/frazane/scoringrules); [Documentation](https://frazane.github.io/scoringrules/)
+	- Reference: [Zanetta and Allen (2024)](https://frazane.github.io/scoringrules/#citation) 	
+
 - **`xskillscore`**
 	- Open source Python package that provides verification metrics of deterministic (and probabilistic from [`properscoring`](https://github.com/properscoring/properscoring)) forecasts with xarray.
 	- [Software Repository](https://github.com/xarray-contrib/xskillscore); [Documentation](https://xskillscore.readthedocs.io/en/latest/)