Handling interruptions for Conversational AI

[duhtapioca]

Hi @freddyaboulton,

With the "send-receive" mode and ReplyOnPause, it looks like when there's an output audio being emitted it's not possible to simultaneously also listen to audio to potentially handle interruptions.

Any advice on how to implement interruption handling to stop the LLM or TTS stream after detecting voice activity? I tried modifying the ReplyOnPause stream handler to process audio while state.responding == True, but I ran into issues as I'm not fully familiar with the code.

For context, I am working with ultravox with XTTSv2 streaming on top of the demo code from your PR in the repo. Any help regarding this would be really appreciated.

Thanks!

[freddyaboulton]

Hi @duhtapioca ! Thanks for the question.

I have not done this myself yet but I wrote some thoughts about how to do this here: #24 (comment)

Let me know how it goes - I'll look into this next week and maybe build something into the library as others are probably having the same question.

[seetimee]

the same need

[VijayRathore24]

Any advice on implementing interruption handling to stop the LLM or TTS stream after detecting voice activity? I tried modifying the ReplyOnPause stream handler to process audio while state. responding == True, but I ran into issues as I'm unfamiliar with the code.

[freddyaboulton]

Hi everyone will be building this into the codebase.