Invalid unicode characters break transcribe

[andrewchen5678]

When it encounters invalid unicode characters, it breaks transcription without a way to workaround it such as replacing invalid unicode with https://codepoints.net/U+FFFD:

  File "/Users/it3/Library/Caches/pypoetry/virtualenvs/speech-detector-5k8eyuOl-py3.12/lib/python3.12/site-packages/pywhispercpp/model.py", line 133, in transcribe
    res = self._transcribe(audio, n_processors=n_processors)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/it3/Library/Caches/pypoetry/virtualenvs/speech-detector-5k8eyuOl-py3.12/lib/python3.12/site-packages/pywhispercpp/model.py", line 248, in _transcribe
    pw.whisper_full(self._ctx, self._params, audio, audio.size)
  File "/Users/it3/Library/Caches/pypoetry/virtualenvs/speech-detector-5k8eyuOl-py3.12/lib/python3.12/site-packages/pywhispercpp/model.py", line 264, in __call_new_segment_callback
    res = Model._get_segments(ctx, start, n)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/it3/Library/Caches/pypoetry/virtualenvs/speech-detector-5k8eyuOl-py3.12/lib/python3.12/site-packages/pywhispercpp/model.py", line 154, in _get_segments
    text = pw.whisper_full_get_segment_text(ctx, i)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 15-16: invalid continuation byte

When it crashes, it hangs the program instead of being able to continue.

[andrewchen5678]

Just some idea about how to fix it by passing the raw byte array back and use python to decode instead, I am not sure if I should open a PR with this change since I am not familar with C and pybind11 https://github.com/absadiki/pywhispercpp/compare/main...andrewtheguy:pywhispercpp:unicode-handling?expand=1

[absadiki]

Can you provide a sample file that raises this exception ?

But Yes please, go ahead and open a new PR. The changes look good, just squash the commits into one or rebase the branch to clean the commits history, plus make sure to ignore the .vscode config files before committing.

[andrewchen5678]

Here you go @absadiki #93

[andrewchen5678]

resolved by #93