diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 39f30ad9..5410bcbe 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -21,5 +21,52 @@ ## Screenshots (if appropriate): +## Metrics (if appropriate): + +If you submitting a PR for a prediction algorithm (segmentation, identification, +or classification) please fill in values for as many as the below statistics as +are relevant. + +*algorithms by metric* + +metric | relevant algorithms +-------|-------------------- +[accuracy 1 2](https://stats.stackexchange.com/a/231237/143678) | classification, identification +[data IO](https://unix.stackexchange.com/questions/55212) | classification, identification, segmentation +[Dice coefficient 3](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) | segmentation +[disk space usage](https://www.cyberciti.biz/faq/linux-check-disk-space-command) | classification, identification, segmentation +[Hausdorff distance 3](https://en.wikipedia.org/wiki/Hausdorff_distance) | segmentation +[Jaccard index](https://en.wikipedia.org/wiki/Jaccard_index) | segmentation +[Log Loss](http://wiki.fast.ai/index.php/Log_Loss) | classification, identification 4 +[memory usage](https://stackoverflow.com/questions/110259) | classification, identification, segmentation +[prediction time 2](https://stackoverflow.com/questions/385408) | classification, identification, segmentation +[sensitivity 3](http://wiki.fast.ai/index.php/Deep_Learning_Glossary#Recall) | segmentation +[specificity 3](http://wiki.fast.ai/index.php/Deep_Learning_Glossary#Specificity) | segmentation +[training time 2](https://stackoverflow.com/questions/385408) | classification, identification, segmentation + +*notes* + +1. Use 5-fold cross validation if there is enough time and computational power available, otherwise use a holdout set +1. This metric may be automatically calculated by the machine learning architecture, e.g., Keras +1. The calculations for these metrics [are available here](https://github.com/concept-to-clinic/concept-to-clinic/blob/master/prediction/src/algorithms/segment/src/evaluate.py) +1. In order to calculate Log Loss for identification, the data needs to be arranged in a way that shows for each pixel, whether or not it is a nodule centriod. Restated, the pixel level labels of 1/0 would correspond to centriod/not-centriod. + +*metrics by algorithm* + +algorithm | relevant metrics +---------------|------------------ +classification | accuracy, data IO, disk space usage, Log Loss, memory usage, prediction time, training time +identification | accuracy, data IO, disk space usage, Log Loss, memory usage, prediction time, training time +segmentation | data IO, Dice coefficient, disk space usage, Hausdorff distance, Jaccard index, memory usage, prediction time, sensitivity, specificity, training time + +When reporting your values, please use a format similar to the following example. + +algorithm | metric | value +-------------|--------|------: +segmentation | accuracy | 99.5 +segmentation | Jaccard index | 0.5 +segmentation | prediction time (s) | 45.3 +segmentation | memory usage (MB) | 5.4 + ## CLA -- [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well \ No newline at end of file +- [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well