Speaker diarization (partitioning audio based on speaker identity)

[dehma01]

Is there a simple way to do speaker diarization with whisper ?

[omenocal]

I'm also interested in the speaker diarization feature.

[TejaswiniiB]

Me too

[HeadStudios]

Yep same here, trying with Hugging Face but it's not working

[actuallyrizzn]

Yeah, the system requirements for it are pretty nuts.

[HeadStudios]

I know right! By the way I started using an API for it which gives 3 hours free - I do not remember the name off the top of my head but it just seems so much easier than what Hugging Face requires. Let me know if you need the direct URL and I can look it up for you - but I imagine if you search 'diarization api' it should come up. Chrs!

[NavodPeiris]

@omenocal @ejkitchen @TejaswiniiB @HeadStudios @actuallyrizzn

checkout this python package created by me: https://pypi.org/project/speechlib/
this is github repo: https://github.com/Navodplayer1/speechlib

you can do transcription, speaker diarization and speaker recognition all together and get a transcript with actual speaker names!

[ShantanuNair]

A common approach to accomplish diarization is to first creating embeddings (think vocal features fingerprints) for each speech segment (think a chunk of speech based you obtain from the timestamps. Something like 10:00s -> 13.52s would be a segment), and then clustering the embeddings based on that, so you know which speech segments can be grouped and assigned a speaker.

I have only glanced through the paper, and will look into it more in depth, but the key focus would be to see if Whisper can create such embeddings. My gut feels this would be unlikely, as the model focuses more on what is said than how something is said.

[NavodPeiris]

checkout this python package created by me: https://pypi.org/project/speechlib/
this is github repo: https://github.com/Navodplayer1/speechlib

you can do transcription, speaker diarization and speaker recognition all together and get a transcript with actual speaker names!

[lmmx]

Example script using pyannote (unrelated software project) for diarisation, grouping consecutive segments by the same speaker into a 'rolled up' dataframe rather than every segment pyannote finds: https://gist.github.com/lmmx/0970a01295e12531f6a3f0ac5537e0b8

Speaker verification example here, using SpeechBrain: https://github.com/pyannote/pyannote-audio/blob/develop/tutorials/speaker_verification.ipynb

Keen to develop a solution that matches up to Whisper here, and I expect @ShantanuNair's answer is better (and would look more like the speaker verification script).

My gut feels this would be unlikely, as the model focuses more on what is said than how something is said.

I agree, I don't think it'd work with Whisper's output as I've seen it group multiple speakers into a single caption.

(Unfortunately I've seen that putting whisper and pyannote in a single environment leads to a bit of a clash between overlapping dependency versions, namely HuggingFace Hub)

[octimot]

(Unfortunately I've seen that putting whisper and pyannote in a single environment leads to a bit of a clash between overlapping dependency versions, namely HuggingFace Hub)

I can confirm.

I tried to still use pyannote with huggingface-hub==0.10.0 (which works fine for whisper) but I get an exception.

[Majdoddin]

See here for a workaround:
#264 (reply in thread)

[NavodPeiris]

checkout this python package created by me: https://pypi.org/project/speechlib/
this is github repo: https://github.com/Navodplayer1/speechlib

you can do transcription, speaker diarization and speaker recognition all together and get a transcript with actual speaker names!

[octimot]

@Navodplayer1 Please stop sending messages on all threads. We all get all the replies you've sent since we're subscribed to the discussion.

Thanks

[octimot]

After digging a bit more on this subject, it feels that timed speaker diarization is only achievable using a hybrid approach.

Using Pyannote (see Majdoddin's work) seems to be a good and fast solution, but adding silences to audio files that will be later fed to Whisper might generate unwanted hallucinations and influence the context of the transcription, especially for non-english transcripts.

jongwook suggested in another discussion to "do a crude form of speaker turn tracking" by using the prompt attribute in the transcription call and inserting a sort of dialogue example to make the model follow the lead (e.g. " - Hey how are you doing? - I'm doing good. How are you?")

Possible approach:

• first transcribe the original audio using Whisper with initial_prompt="- Hey how are you doing? - I'm doing good. How are you?")
• use pyannote on the same original audio to detect the speakers
• use pyannote's results (timings) to check the segments generated by whisper and confirm the speakers
• then finally provide a report with the pyannote timings that weren't paired with any whisper segments

[Majdoddin]

Meanwhile, I have updated my work, no more silence adding. Check the updated Colab notebook and the result.
And, please consider supporting my work with a donation.

[sonovice]

I tried your approach, @octimot. It works perfectly, thanks.

[hkaraoguz]

I used the proposed approach in this video https://www.youtube.com/watch?v=dUcVcxfk-gc but I couldn't get a good segmentation from whisper using initial_prompt="- Hey how are you doing? - I'm doing good. How are you?" The segments given by whisper without using initial_prompt looked more promising to match with pyannote segments. Is the given initial_prompt generic or scenario specific? Thank you. (I used the base model with whisper 1.0 installed)

[karim-alweheshy]

I found better results when the initial_prompt was relevant to the situation of the audio track

[NavodPeiris]

checkout this python package created by me: https://pypi.org/project/speechlib/
this is github repo: https://github.com/Navodplayer1/speechlib

you can do transcription, speaker diarization and speaker recognition all together and get a transcript with actual speaker names!

[NunoSempere]

Any chance that this diarization could be implemented as a standalone package, rather than having to rely on Google Colab?

[TheRealAlexV]

Would also love this

[NavodPeiris]

checkout this python package created by me: https://pypi.org/project/speechlib/
this is github repo: https://github.com/Navodplayer1/speechlib

you can do transcription, speaker diarization and speaker recognition all together and get a transcript with actual speaker names!

[nhtkid]

Example script using pyannote (unrelated software project) for diarisation, grouping consecutive segments by the same speaker into a 'rolled up' dataframe rather than every segment pyannote finds: https://gist.github.com/lmmx/0970a01295e12531f6a3f0ac5537e0b8

Speaker verification example here, using SpeechBrain: https://github.com/pyannote/pyannote-audio/blob/develop/tutorials/speaker_verification.ipynb

Keen to develop a solution that matches up to Whisper here, and I expect @ShantanuNair's answer is better (and would look more like the speaker verification script).

My gut feels this would be unlikely, as the model focuses more on what is said than how something is said.

I agree, I don't think it'd work with Whisper's output as I've seen it group multiple speakers into a single caption.

(Unfortunately I've seen that putting whisper and pyannote in a single environment leads to a bit of a clash between overlapping dependency versions, namely HuggingFace Hub)

Hey, does that mean you are just gonna use Pyannote to do the transcription instead of Whisper? So it will be no OpenAI involved?

[Majdoddin]

www.lexicaps.com seamlessly adds diarization to Whispers transcription. No 3rd party packages.
Announcement: #1537
Repo: https://github.com/Majdoddin/lexicaps