Debugging USB interrupts that sometimes take a bit too long

[szszoke]

Hello,

I'm using this library in a user space Linux driver for a DSP that uses a custom protocol.

I have a working prototype that essentially exposes the DSP as a PipeWire node.

My prototype suffers from buffer underflows which I think are caused by the USB data transfer sometimes taking a bit too long.

The source code is not yet public but this is what I'm doing in a nutshell:

The DSP has both inputs and outputs, and data for both is transmitted via interrupt endpoints.

I'm using queues for both transmitting and receiving data.

My program needs to send and receive data from both endpoints for the DSP to start sending audio data.

The first version of my program used tokio::select! to wait on next_complete() for both the RX and TX queue.

This worked but the time between each transfer cycle fluctuated a lot.

The DSP accumulates samples for 3.5 milliseconds. I could complete transfer cycles quite close to that figure but often I would observe cycle times that were a few hundred microseconds longer than that.

Sometimes I would get peaks of 4-5 milliseconds or more.

I ended up rewriting my logic to use futures_lite and the times for transfer cycles became more consistent. They were closer to the theoretical minimum of 3.5ms but I would still observe quite many slower cycles.

In this version I have a loop where I block on the two futures returned by the RX and TX queues. I wrap the two results in enum fields and then combine those two futures into one with or from futures_lite (so that I know which of the two transfers completed).

I measure the time between each cycle by reusing an Instant and then comparing the value of elapsed() with the value from the previous transfer cycle.

Do you have any suggestions for using your library in a low latency application such as this?

[szszoke]

One more thing to note is that this issue doesn't seem to be related to CPU usage and it also happens with SMT/HyperThreading disabled.

I also tried to give RT (-80) priority to the thread created by nusb but that didn't seem to help either.

[szszoke]

Here is a code snippet that I am using for testing:

Checking for killswitch.is_triggered() adds about 150 nanoseconds to the transfer cycle time. Other than that I sometimes get a cycle time that is very close to 3500 microseconds, but most take 100-200 microseconds more than that.

let start = Instant::now();
let slow_usb_threshold_micros: u128 = 3500;
let mut rx_cycle_duration: u128 = 0;

while !killswitch.is_triggered() {
  let completion = block_on(rx_queue.next_complete());

  let elapsed = start.elapsed().as_micros();
  let cycle_duration = elapsed - rx_cycle_duration;

  rx_queue.submit(RequestBuffer::reuse(
    completion.data,
    self.usb_rx_buffer_size,
  ));

  if rx_cycle_duration > 0 && cycle_duration > slow_usb_threshold_micros {
    println!(
      "[USB] Slow RX cycle: {}us, diff {}us",
      cycle_duration,
      cycle_duration - slow_usb_threshold_micros,
    );
  }

  rx_cycle_duration = start.elapsed().as_micros();
}

[kevinmehall]

If you aren't already, my first suggestion for the IN endpoint would be to pre-submit several (maybe 2-8) transfers such that there are always transfers pending in the kernel and host controller even if the userspace code doesn't get scheduled, and re-submit them as they complete like your code above.

On the OUT endpoint you could also buffer multiple transfers, but in that direction the data must be passed at the time the transfer is submitted so it would add to latency, but may gain consistency and reliability.

[kevinmehall]

As far as nusb goes, it has a background thread that epolls the USB file descriptor, receives transfer completions, and wakes the executor that is waiting on the relevant queue. That cross-thread wake is handled by the waker implementation in the executor (futures_lite::block_on in your case).

Beyond that, it's mostly in the kernel and out of nusb's control.