Merge pull request #56 from itk-dev/feature/wer-and-cer

3753: WER and CER qualification of donations

CHANGELOG.md

@@ -8,6 +8,10 @@ See [keep a changelog] for information about writing changes to this log.
 
 ## [Unreleased]
 
+* [PR-56](https://github.com/itk-dev/doner-din-stemme/pull/56)
+  * Modifies qualify command to only transcribe donations.
+  * Makes commands calculating each of the metrics (WER, CER and similar_text)
+
 ## [1.0.0] - 2024-11-29
 
 * [PR-55](https://github.com/itk-dev/giv-din-stemme/pull/55)
@@ -79,7 +83,6 @@ See [keep a changelog] for information about writing changes to this log.
   Mobile optimization.
 * Initial project.
 
-[keep a changelog]: https://keepachangelog.com/en/1.1.0/
-[Unreleased]: https://github.com/itk-dev/doner-din-stemme/compare/main...develop
-[//]: # ([1.0.1]: https://github.com/itk-dev/doner-din-stemme/compare/1.0.0...1.0.1)
+[Unreleased]: https://github.com/itk-dev/doner-din-stemme/compare/1.0.0...HEAD
 [1.0.0]: https://github.com/itk-dev/doner-din-stemme/releases/tag/1.0.0
+[keep a changelog]: https://keepachangelog.com/en/1.1.0/

README.md

@@ -211,24 +211,42 @@ The custom Giv din stemme module adds commands using
 [Whisper](https://github.com/itk-dev/whipser-docker) for qualifying donations.
 
 Qualifying is done by asking Whisper to transcribe the donation and then
-comparing it to the original text using PHPs
-[similar_text](https://www.php.net/manual/en/function.similar-text.php).
+comparing it to the original text, using one of three metrics.
+
+### Similar text score
+
+[similar_text](https://www.php.net/manual/en/function.similar-text.php) gives similarity
+between two sentences as a percentage. That is, a high percentage means the sentences are similar.
+See [ADR-002](/docs/adr/002-whisper-score-via-similar-text.md) for more details.
+
+**Note**: to align with the WER and CER scores beneath, we report the complimentary,
+i.e. the dissimilarity, on the list of donations.
+
+### WER and CER
+
+Word error rate(WER) and character error rate(CER). For more details on these, see
+[itk-dev/sentence-similarity-metrics](https://github.com/itk-dev/sentence-similarity-metrics?tab=readme-ov-file#metrics).
+This is a number between 0 and 1  A low score means the transcribed sentence is similar to the original one.
 
 ### Configuration
 
 Before using the qualifying command you must configure
 
 * Whisper API endpoint
 * Whisper API key
-* Threshold for when donations should be automatically validated (int or null/unset to disable)
+* Similar text score threshold for when donations should be validated (int or null/unset to disable).
+* WER threshold for when donations should be validated (float or null/unset to disable).
+* CER threshold for when donations should be validated (float or null/unset to disable).
 
 ```php
 // settings.local.php
 // …
 
 $settings['itkdev_whisper_api_endpoint'] = '…';
 $settings['itkdev_whisper_api_key'] = '…';
-$settings['itkdev_automatic_validation_threshold'] = 90;
+$settings['itkdev_automatic_validation_threshold_similar_text_score'] = 80;
+$settings['itkdev_automatic_validation_threshold_wer'] = 0.20;
+$settings['itkdev_automatic_validation_threshold_cer'] = 0.20;
 ```
 
 See 1Password for both api endpoint and key.
@@ -237,24 +255,50 @@ See 1Password for both api endpoint and key.
 
 Qualify all unqualified donations with
 
-```shell name="gds-qualify-all-donations"
-itkdev-docker-compose drush giv_din_stemme:qualify:all
+```shell name="gds-qualify-transcribe-all-donations"
+itkdev-docker-compose drush giv_din_stemme:qualify:transcribe
 ```
 
 or re-qualify donations by adding the `--re-qualify` flag.
 
 Qualify a specific donation with
 
-```shell name="gds-qualify-specific-donations"
-itkdev-docker-compose drush giv_din_stemme:qualify:donation DONATION_ID
+```shell name="gds-qualify-transcribe-specific-donation"
+itkdev-docker-compose drush giv_din_stemme:qualify:transcribe:id DONATION_ID
 ```
 
-**Note** that both qualifying commands will validate donations if they result
-in a `similar_text` score that exceeds the configured threshold level.
+**Note** the `similar_text` qualifying command will validate donations
+if they result in a score that surpasses the configured threshold level.
+The `wer` and `cer` will validate the donation if the score does not
+surpass the configured threshold levels.
 The commands will never invalidate donations.
 
-To continuously qualify donations consider running the qualify all donations
-command via a cronjob.
+Calculate similar text score with
+
+```shell name="gds-qualify-similar-text-score"
+itkdev-docker-compose drush giv-din-stemme:qualify:similar-text-score
+```
+
+or re-calculate rates by adding the `--re-calculate` flag.
+
+Calculate WER with
+
+```shell name="gds-qualify-wer"
+itkdev-docker-compose drush giv-din-stemme:qualify:wer
+```
+
+or re-calculate rates by adding the `--re-calculate` flag.
+
+Calculate CER with
+
+```shell name="gds-qualify-cer"
+itkdev-docker-compose drush giv-din-stemme:qualify:cer
+```
+
+or re-calculate rates by adding the `--re-calculate` flag.
+
+To continuously qualify donations consider running the qualifying commands
+via a cronjobs.
 
 ## Coding standards
 

composer.json

@@ -20,6 +20,7 @@
         "drupal/pathauto": "^1.13",
         "drupal/twig_tweak": "^3.4",
         "drush/drush": "^13.0",
+        "itk-dev/sentence-similarity-metrics": "^0.1.0",
         "itk-dev/web_accessibility_statement": "dev-feature/drupal-11-support"
     },
     "require-dev": {

docs/adr/003-additional-whisper-scores-wer-and-cer.md

@@ -0,0 +1,24 @@
+# Donation pre-qualification via WER and CER
+
+Date: 22-10-2024
+
+## Status
+
+Accepted
+
+## Context
+
+Word error rate (WER) and character error rate (CER) are common metrics used
+when measuring the performance of automated speech recognition system.
+
+## Decision
+
+In addition to the `similar_text` score discussed in [ADR-002](002-whisper-score-via-similar-text.md)
+WER and CER should be incorporated in the dataset. These should also allow automatic validation.
+
+For more details on qualification of donations see [https://github.com/itk-dev/giv-din-stemme?tab=readme-ov-file#whisper](https://github.com/itk-dev/giv-din-stemme?tab=readme-ov-file#whisper)
+
+## Consequences
+
+Be aware that previously validated donations will have been validated based on `similar_text` scores.
+If we simply wish to validate based on WER or CER we might have to invalidate previous donations.

web/modules/custom/giv_din_stemme/giv_din_stemme.install

@@ -37,3 +37,22 @@ function giv_din_stemme_update_11002() {
   \Drupal::entityDefinitionUpdateManager()
     ->installFieldStorageDefinition('whisper_guess_similar_text_score', 'gds', 'gds', $similarTextScoreFieldDefinition);
 }
+
+/**
+ * Add WER and CER fields to gds table.
+ */
+function giv_din_stemme_update_11003() {
+  $wordErrorRateFieldDefinition = BaseFieldDefinition::create('float')
+    ->setLabel(t('Word error rate'))
+    ->setDescription(t('Word error rate between original text and whisper guess'));
+
+  \Drupal::entityDefinitionUpdateManager()
+    ->installFieldStorageDefinition('whisper_guess_word_error_rate', 'gds', 'gds', $wordErrorRateFieldDefinition);
+
+  $characterErrorRateFieldDefinition = BaseFieldDefinition::create('float')
+    ->setLabel(t('Character error rate'))
+    ->setDescription(t('Character error rate between original text and whisper guess'));
+
+  \Drupal::entityDefinitionUpdateManager()
+    ->installFieldStorageDefinition('whisper_guess_character_error_rate', 'gds', 'gds', $characterErrorRateFieldDefinition);
+}