silence detection causes audio issues with short sounds

[totaam]

Issue migrated from trac ticket # 2855

component: server | priority: major

2020-08-03 09:44:22: pot created the issue

---

Sound is generally not reliable for short sounds.

If I play music or something longer than one second things are generally good (apart, sometimes, from the very start of sound). But for beeps or short sounds the sound is randomly broken. Tried with opus and flac, with or without av-sync, without any apparent difference.

For example, this sequence of beeps always come out broken, every time in a different way (you need Sox to play this):

play -n synth 0.3 \
    trapezium 1000 0.3 trapezium 1000 0.3 \
    trapezium 1000 0.525 trapezium 2000 0.525 \
    trapezium 1500 delay 2 +.45 +.45 +.45 +.45 remix - gain -n -6 repeat 2

This is the output of an start command followed by attach, issued from the client

Warning: vendor 'Intel' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-03 10:20:14,836 Xpra GTK3 X11 client version 4.0.2-26625 64-bit
2020-08-03 10:20:14,980  running on Linux Debian unstable sid
2020-08-03 10:20:14,981  window manager is 'Metacity (Marco)'
2020-08-03 10:20:14,998 Warning: failed to import opencv:
2020-08-03 10:20:14,999  No module named 'cv2'
2020-08-03 10:20:14,999  webcam forwarding is disabled
2020-08-03 10:20:15,266 GStreamer version 1.16.2 for Python 3.8.5 64-bit
2020-08-03 10:20:15,402 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-03 10:20:15,556 Warning: vendor 'Intel' is greylisted,
2020-08-03 10:20:15,556  you may want to turn off OpenGL if you encounter bugs
2020-08-03 10:20:15,612 OpenGL enabled with Mesa Intel(R) Iris(R) Plus Graphics 655 (CFL GT3)
2020-08-03 10:20:15,870 Connected (version 2.0, client OpenSSH_8.3p1)
2020-08-03 10:20:16,044 Authentication (publickey) failed.
2020-08-03 10:20:16,093 Authentication (publickey) successful!
2020-08-03 10:20:16,323  keyboard settings: rules=evdev, model=pc105, layout=us,it
2020-08-03 10:20:16,340  desktop size is 1280x1024 with 1 screen:
2020-08-03 10:20:16,341   :0.0 (339x271 mm - DPI: 95x95) workarea: 1279x996 at 0x28
2020-08-03 10:20:16,342     GSM DP-1 (376x301 mm - DPI: 86x86)
Warning: vendor 'Intel' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-03 10:20:19,595 Xpra GTK3 X11 client version 4.0.2-26625 64-bit
2020-08-03 10:20:19,738  running on Linux Debian unstable sid
2020-08-03 10:20:19,739  window manager is 'Metacity (Marco)'
2020-08-03 10:20:19,757 Warning: failed to import opencv:
2020-08-03 10:20:19,758  No module named 'cv2'
2020-08-03 10:20:19,758  webcam forwarding is disabled
2020-08-03 10:20:20,030 GStreamer version 1.16.2 for Python 3.8.5 64-bit
2020-08-03 10:20:20,110 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-03 10:20:20,266 Warning: vendor 'Intel' is greylisted,
2020-08-03 10:20:20,267  you may want to turn off OpenGL if you encounter bugs
2020-08-03 10:20:20,325 OpenGL enabled with Mesa Intel(R) Iris(R) Plus Graphics 655 (CFL GT3)
2020-08-03 10:20:20,538 Connected (version 2.0, client OpenSSH_8.3p1)
2020-08-03 10:20:20,712 Authentication (publickey) failed.
2020-08-03 10:20:20,761 Authentication (publickey) successful!
2020-08-03 10:20:21,082  keyboard settings: rules=evdev, model=pc105, layout=us,it
2020-08-03 10:20:21,083  desktop size is 1280x1024 with 1 screen:
2020-08-03 10:20:21,083   :0.0 (339x271 mm - DPI: 95x95) workarea: 1279x996 at 0x28
2020-08-03 10:20:21,084     GSM DP-1 (376x301 mm - DPI: 86x86)
2020-08-03 10:20:27,641 enabled remote logging
2020-08-03 10:20:27,643 Xpra GTK3 X11 server version 4.0.2-26625 64-bit
2020-08-03 10:20:27,643  running on Linux Debian testing bullseye
2020-08-03 10:20:27,655 Attached to ssh://[email protected]/7
2020-08-03 10:20:27,655  (press Control-C to detach)

2020-08-03 10:20:27,762 server does not support xi input devices
2020-08-03 10:20:27,762  server uses: xtest
2020-08-03 10:20:27,910 sound output using pulseaudio device:
2020-08-03 10:20:27,911 sound output  'Built-in Audio Digital Stereo (HDMI)'
2020-08-03 10:20:28,429 sound output using 'opus' audio codec
/usr/lib/python3/dist-packages/gi/overrides/Gtk.py:1632: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  return _Gtk_main(*args, **kwargs)
2020-08-03 10:20:30,403 enabled remote logging
2020-08-03 10:20:30,404 Xpra GTK3 X11 server version 4.0.2-26625 64-bit
2020-08-03 10:20:30,404  running on Linux Debian testing bullseye
2020-08-03 10:20:30,515 server does not support xi input devices
2020-08-03 10:20:30,515  server uses: xtest
2020-08-03 10:20:30,664 sound output using pulseaudio device:
2020-08-03 10:20:30,664 sound output  'Built-in Audio Digital Stereo (HDMI)'
2020-08-03 10:20:31,183 sound output using 'opus' audio codec
2020-08-03 10:20:32,343 server requested disconnect:
2020-08-03 10:20:32,343  new client
2020-08-03 10:20:32,343  new connection from the same uuid

[totaam]

I believe you must be experiencing side-effects from #1617.
If that's the case, you can disable the "silence detection" by starting your server using:

xpra start --env=CUTTER_THRESHOLD=0 ...

If that's the cause, this will show more information on what it is doing:

XPRA_SOUND_LOG_CUTTER=1

And if that's the case, then I will close as 'notabug' since this is a feature and there is a workaround.

[totaam]

2020-08-03 10:24:46: pot commented

---

The problem is still there: some parts of the sound are lost, some are delayed.

Using CUTTER_THRESHOLD=0 I see some improvement, in that the number of losses and delays is roughly halved.

What next? Should I try both CUTTER_THRESHOLD=0 and XPRA_SOUND_LOG_CUTTER=1 ?

[totaam]

2020-08-04 14:50:32: pot commented

---

Some news: I tried with and without av-sync, with and without CUTTER_THRESHOLD=0, with opus, mp3, flac, wav.

The only thing that works is wav, with or withoout av-sync and CUTTER_THRESHOLD=0. I occasionally still hear a broken beep, but that is rare. This is my current configuration. If someone tells me how to debug this, I can try.

[totaam]

I tried with and without av-sync
I should have explained: av-sync has no effect on audio, only video.

The only thing that works is wav, with or withoout av-sync and CUTTER_THRESHOLD=0.
Sorry, my mistake: the correct spelling for all xpra related environment variables is XPRA_xxxx, ie:

XPRA_CUTTER_THRESHOLD=0

The reason why this still worked for wav is because wav and a few others disable cutter:

xpra/sound/gstreamer_util.py:ENCODER_CANNOT_USE_CUTTER = ("vorbisenc", "wavpackenc", "avenc_aac", "wavenc")

With the correct spelling, you should be able to use the more efficient codecs, in particular opus.

[totaam]

2020-08-06 11:53:15: pot commented

---

Okay, I can confirm that --env=XPRA_CUTTER_THRESHOLD=0 hides the problem and the sound comes out well with the opus encoder.

Should I report anything about what it happens?
If yes, should I set both --env=XPRA_CUTTER_THRESHOLD=0 and --env=XPRA_SOUND_LOG_CUTTER=1?

[totaam]

The savings from silence detection are huge and we have a viable workaround, so I this issue is very unlikely to get fixed anytime soon

[totaam]

2020-08-06 14:10:44: pot commented

---

Ok, I understand, thank you for your excellent work.

But this is not an answer to my question :)

[totaam]

But this is not an answer to my question :)

Should I report anything..

No, I'm sure the problem is easily reproducible, XPRA_SOUND_LOG_CUTTER=1 doesn't help on its own.
This would need investigating, and I don't have the time.

[totaam]

We have switched to removesilence in v5: #3709, this should solve these glitches.
Please re-open if not.