[runtime/gpu] Add GPU Hotwords

runtime/gpu/hotwords/README.md

@@ -0,0 +1,137 @@
+## Introduction
+The below example shows how to deploy WeNet ASR offline or streaming models with hotwordsboosting on GPUs.
+
+## Instructions
+* Step 1. Convert your pretrained model to onnx models
+```bash
+conda activate wenet
+pip install onnxruntime-gpu onnxmltools
+cd wenet/examples/aishell2/s0 && . ./path.sh
+
+# offline model
+wget https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20210601_u2%2B%2B_conformer_exp.tar.gz --no-check-certificate
+tar -zxvf 20210601_u2++_conformer_exp.tar.gz
+model_dir=$(pwd)/20210601_u2++_conformer_exp
+
+# streaming model
+wget https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20210601_u2%2B%2B_conformer_exp.tar.gz --no-check-certificate
+tar -zxvf 20210601_u2++_conformer_exp.tar.gz
+model_dir=$(pwd)/20210601_u2++_conformer_exp
+
+onnx_model_dir=$model_dir/onnx_model_dir
+mkdir $onnx_model_dir
+
+# offline model
+python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.3 --reverse_weight=0.3 --output_onnx_dir=$onnx_model_dir --fp16
+
+# streaming model
+python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.3 --reverse_weight=0.3 --output_onnx_dir=$onnx_model_dir --fp16 --streaming
+
+cp $model_dir/units.txt $onnx_model_dir
+cp $model_dir/units.txt $onnx_model_dir/words.txt
+cp $model_dir/train.yaml $onnx_model_dir/
+```
+
+* Step 2. Copy hotwords related files to onnx_model_dir folder
+```bash
+cd wenet/runtime/gpu/hotwords
+cp hotwords.yaml $onnx_model_dir
+```
+
+* Step 3. Build server docker and start server on one gpu
+```
+cd wenet/runtime/gpu
+# docker version >= 19.03.2
+# If download fails, please use https://huggingface.co/58AILab/wenet_u2pp_aishell1_with_hotwords/blob/main/Dockerfile/Dockerfile.hotwordsserver
+docker build . -f Dockerfile/Dockerfile.server -t wenet_hotwords_server:latest --network host
+
+# offline model
+docker run --gpus '"device=0"' --rm -it -v $PWD:/ws/gpu -v $PWD/hotwords/model_repo_hotwords:/ws/gpu/hotwords/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1 wenet_hotwords_server:latest /workspace/scripts/convert_start_hotwords_server.sh
+
+# streaming model
+docker run --gpus '"device=0"' --rm -it -v $PWD:/ws/gpu -v $PWD/hotwords/model_repo_stateful_hotwords:/ws/gpu/hotwords/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1  wenet_hotwords_server:latest /workspace/scripts/convert_start_hotwords_server.sh
+```
+
+* Step 4. Start client
+```
+cd wenet/runtime/gpu
+docker build . -f Dockerfile/Dockerfile.client -t wenet_client:latest --network host
+AUDIO_DATA=$PWD/client/test_wavs
+docker run -it --net host --name wenet_hotwords_client -v $PWD/client:/ws/client -v $AUDIO_DATA:/ws/test_data wenet_client:latest
+
+# In docker
+# offline model test
+cd /ws/client
+
+# test one wav file
+python3 client.py --audio_file=/ws/test_data/mid.wav --url=localhost:8001
+
+# test a list of wav files & cer
+python3 client.py --wavscp=/ws/dataset/test/wav.scp --data_dir=/ws/dataset/test/ --trans=/ws/dataset/test/text
+
+# streaming model test
+python3 client.py --audio_file=/ws/test_data/mid.wav --url=localhost:8001 --model_name=streaming_wenet --streaming
+```
+
+## Hotwords Test
+
+Base Acoustic model: [20210601_u2++_conformer_exp (AISHELL-1)](https://github.com/wenet-e2e/wenet/blob/main/docs/pretrained_models.md)
+
+Tested ENV
+* CPU：40 Core, Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
+* GPU：NVIDIA GeForce RTX 2080 Ti
+
+Hotwords file: https://huggingface.co/58AILab/wenet_u2pp_aishell1_with_hotwords/tree/main/models
+
+[AISHELL-1 Test dataset](https://www.openslr.org/33/)
+
+* Test set contains 7176 utterances (5 hours) from 20 speakers.
+
+| model (FP16)                 | RTF     | CER    |
+|------------------------------|---------|--------|
+| offline model w/o hotwords   | 0.00437 | 4.6805 |
+| offline model w/  hotwords   | 0.00428 | 4.5841 |
+| streaming model w/o hotwords | 0.01231 | 5.2777 |
+| streaming model w/  hotwords | 0.01195 | 5.1850 |
+
+[AISHELL-1 hostwords sub-testsets](https://www.modelscope.cn/datasets/speech_asr/speech_asr_aishell1_hotwords_testsets/summary)
+
+* Test set contains 235 utterances with 187 entities words.
+
+| model (FP16)               | Latency (s) | CER   | Recall | Precision | F1-score |
+|----------------------------|-------------|-------|--------|-----------|----------|
+| offline model w/o hotwords | 5.8673      | 13.85 | 0.27   | 0.99      | 0.43     |
+| offline model w/  hotwords | 5.6601      | 11.96 | 0.47   | 0.97      | 0.63     |
+
+Decoding result
+
+| Label                | hotwords  | pred w/o hotwords            | pred w/ hotwords             |
+|----------------------|-----------|------------------------------|------------------------------|
+| 以及拥有陈露的女单项目          | 陈露        | 以及拥有**陈鹭**的女单项目              | 以及拥有**陈露**的女单项目              |
+| 庞清和佟健终于可以放心地考虑退役的事情了 | 庞清<br/>佟健 | **庞青**和**董建**终于可以放心地考虑退役的事情了 | **庞清**和**佟健**终于可以放心地考虑退役的事情了 |
+| 赵继宏老板电器做厨电已经三十多年了    | 赵继宏       | **赵继红**老板电器做厨店已经三十多年了        | **赵继宏**老板电器做厨电已经三十多年了        |
+
+Refer to more results: https://huggingface.co/58AILab/wenet_u2pp_aishell1_with_hotwords/tree/main/results
+
+## Hotwords usage
+Please refer to the following steps how to use hotwordsboosting.
+* Step 1. Initialize HotWordsScorer
+```
+# if you don't want to use hotwords. set hotwords_scorer=None(default),
+# vocab_list is Chinese characters.
+hot_words = {'再接': 10, '再厉': -10, '好好学习': 100}
+hotwords_scorer = HotWordsScorer(hot_words, vocab_list, is_character_based=True)
+```
+If you set is_character_based is True (default mode), the first step is to combine Chinese characters into words, if words in hotwords dictionary then add hotwords score. If you set is_character_based is False, all words in the fixed window will be enumerated.
+
+* Step 2. Add hotwords_scorer when decoding
+```
+result = ctc_beam_search_decoder_batch(batch_chunk_log_prob_seq,
+                                        batch_chunk_log_probs_idx,
+                                        batch_root_trie,
+                                        batch_start,
+                                        beam_size, num_processes,
+                                        blank_id, space_id,
+                                        cutoff_prob, scorer, hotwords_scorer)
+```
+Please refer to [swig/test/test_zh.py](https://github.com/Slyne/ctc_decoder/blob/master/swig/test/test_zh.py#L108) for how to decode with hotwordsboosting.

runtime/gpu/hotwords/hotwords.yaml

@@ -0,0 +1,187 @@
+邓郁松: 100
+王颖春: 100
+布赖恩克尔扎尼奇: 100
+王晔君: 100
+许玮甯: 100
+阮经天: 100
+朱圣祎: 100
+王思聪: 100
+宋芳: 100
+刘澄: 100
+陈昊芝: 100
+阿拉木图: 100
+陈露: 100
+平昌: 100
+许绍洋: 100
+金陵: 100
+陈传钧: 100
+东莞: 100
+蒲东峰: 100
+杨锋: 100
+李斯达: 100
+周云露: 100
+阳光嘉园: 100
+罗宾索尔科维: 100
+萨维申科: 100
+隋文静: 100
+韩聪: 100
+彭程: 100
+张昊: 100
+马塞洛特: 100
+孙楠: 100
+巫启贤: 100
+许茹芸: 100
+崔栽诚: 100
+刘玮: 100
+李玉刚: 100
+张学友: 100
+黄琦雯: 100
+陆骑麟: 100
+徐云丽: 100
+徐永昊: 100
+科林哈迪: 100
+诸葛紫岐: 100
+谢依霖: 100
+顾欣: 100
+高新区钓渭镇疙瘩沟村: 100
+王晓飞: 100
+包尔江: 100
+大兴区礼贤镇紫各庄村: 100
+王巍: 100
+大兴区黄村镇狼垡三村: 100
+张绵跃: 100
+奉化江口儒江村: 100
+本加卢鲁: 100
+施王祥: 100
+南粤: 100
+大禹村: 100
+黎胜明: 100
+淅川县: 100
+灵璧县: 100
+吴雯: 100
+张锦麟: 100
+佟健: 100
+庞清: 100
+刘国梁: 100
+杨爱明: 100
+留光镇: 100
+诏安县: 100
+雁翅镇: 100
+李冬梅: 100
+张贵林: 100
+伟嘉安捷: 100
+赵伟国: 100
+关锡友: 100
+张蓉芳: 100
+普鲁申科: 100
+卢民相: 100
+朴泰桓: 100
+高桥大辅: 100
+李焜染: 100
+黄欣晨: 100
+佟继萍: 100
+王亦菲: 100
+张诗欢: 100
+清澜港: 100
+孙永杰: 100
+浅田真央: 100
+小林尊: 100
+崔帅豪: 100
+泰舒培: 100
+黄榕: 100
+盐城大分港: 100
+王晓宇: 100
+蚌飞市: 100
+湘龙西路: 100
+杨幼萍: 100
+弗菜戈: 100
+陈嘉宝: 100
+赖慰玲: 100
+陈凯琳: 100
+郑嘉颖: 100
+田心妮: 100
+张保仔: 100
+洪永城: 100
+项锐: 100
+黄梦思: 100
+嵩山: 100
+齐永信: 100
+龙港镇: 100
+邬雄: 100
+郎平: 100
+洪永城: 100
+陈展鹏: 100
+苏迪罗: 100
+范跃红: 100
+瓯文: 100
+玉海: 100
+富平: 100
+张淑侠: 100
+许胚: 100
+莒溪大峡谷: 100
+哈立: 100
+祖耶夫: 100
+雅桑克莱: 100
+陈势安: 100
+王靖苏: 100
+麻丽贤: 100
+龙湾永强: 100
+邬迪: 100
+鄂美蝶: 100
+大滨芳美: 100
+伊萨: 100
+张春雨: 100
+刘艾林: 100
+黄煜坤: 100
+张晓雅: 100
+卢加诺: 100
+欧米茄: 100
+彼得许尔泽勒: 100
+莎宾娜: 100
+常平: 100
+刘家义: 100
+万龙: 100
+云顶: 100
+海坨山: 100
+刘庆红: 100
+朱祖儿: 100
+潘玮柏: 100
+张呈亮: 100
+赵书伶: 100
+武红利: 100
+陈晟: 100
+美芝: 100
+威灵: 100
+付敬懿: 100
+李晖: 100
+海珠湖: 100
+杨德合: 100
+喻莉: 100
+胡琦: 100
+乌守保: 100
+丁霞: 100
+刘晏含: 100
+沈静思: 100
+张常宁: 100
+惠若琪: 100
+科舍列娃: 100
+晋吉岛: 100
+西安: 100
+吴显亭: 100
+东莞: 100
+顺德: 100
+张小飞: 100
+多米尼加: 100
+马丁内斯: 100
+朱婷: 100
+敦煌: 100
+李洁: 100
+明孝陵: 100
+贵港: 100
+钟欣: 100
+吴文新: 100
+赵继宏: 100
+颜妮: 100
+桃太郎: 100
+陈观鑫: 100
+新洲: 100

runtime/gpu/hotwords/model_repo_hotwords/attention_rescoring

@@ -0,0 +1 @@
+../../model_repo/attention_rescoring

runtime/gpu/hotwords/model_repo_hotwords/decoder

@@ -0,0 +1 @@
+../../model_repo/decoder

runtime/gpu/hotwords/model_repo_hotwords/encoder

@@ -0,0 +1 @@
+../../model_repo/encoder

runtime/gpu/hotwords/model_repo_hotwords/feature_extractor

@@ -0,0 +1 @@
+../../model_repo/feature_extractor