Audio Interruption: Assistant Silent or Unresponsive with fastrtc

[albertofh98]

Hi!

In order to handle audio interruptions with my real-time OpenAI application via fastrtc library, I simply need to:

# In my case, I simply declared webrtc as a global variable
elif event.type == "input_audio_buffer.speech_started":
  k = list(webrtc.connections.keys())[0]
  audio_callback = webrtc.connections[k][0]
  
  while not audio_callback.queue.empty():
      # Depending on your program, you may want to
      # catch QueueEmpty
      audio_callback.queue.get_nowait()
      audio_callback.queue.task_done()

However, while the previous implementation for managing interruptions generally works, I've encountered an issue where, after some interruptions, the assistant remains silent for several seconds before responding with the new answer, despite the audio queue being cleared and new audio chunks being processed for the next response. Additionally, there are instances where the assistant does not respond at all after an interruption, failing to play any further audio.

What do you think could be causing this?

[freddyaboulton]

Hi @albertofh98 - I think clearing the queue may be messing up the timestamps of the frames, causing it to pause. Line here.

Can you also set self._start = None ? I will build this into the library