In-Game: Decouple audio-decoding threads from the audio context

[toby-yoyo]

Is your feature request related to a problem?

The primary aims of the audio decoding threads are:

• To decode a given source of compressed audio data.
• To seek around the source when requested.
• To provide some information about the codec state (stream position etc.).

Our decoding threads do achieve these things. However, they also do some other things, such as:

• Buffering decoded audio data.
• Queueing/Unqueueing buffers on the audio context.
• Getting/Setting source states.

The result of this is that many simple processes in the audio engine (e.g. stopping a sound) become an async process where we have to defer actions to the decoding threads. On top of this, we need to present these async processes as synchronous to users, so we have to build up rather complex systems to achieve this.

Additionally, the decoding threads have a few problematic behaviours such as 'detecting' buffer underruns, leading to potentially incorrect source state overrides, and partial deinitialisation of voice subsystems, which steps on the feet of the main thread. There is also a tight coupling of the stream states and source states which doesn't make much sense as these are two mostly unrelated systems.

Finally, we have at minimum 6 different threads that access the audio context (plus more depending on other features that users may/may not be utilising). This presents problems such as increased contention for the context (which is handled by only a single mutex), and loss of error state (which is intended to be read from a single thread).

Describe the solution you'd like

The decoding threads already do the main things they are supposed to, and generally in a reasonable way as well. However, moving the responsibility of certain tasks to the main/audio threads will help to untangle the code somewhat.

We should decouple the decoding threads entirely from the audio context. Decoding threads should not really need to know about or interact with the audio context, or even the engine more generally as they are essentially a specialised file/stream reader.

Specific changes we would make here would include:

• Deferring buffering and queueing of data to the audio thread.
• Consolidation of various distributed state flags into a concrete set of voice states, including an initial 'preparing' and final 'ramping' state.
• Allowing the audio thread to make certain voice state transitions (e.g. preparing->playing and ramping->stopped).
• Enabling voices to detach from decoding tasks, allowing them to be reused without waiting for the decoding task to deinitialise.
• Wrapping the decoding threads/tasks in a generic interface to allow us to integrate new codecs more easily.
• Implementing OpenAL's source types, which provide information about how a source's buffers are provided.

Additional context

This isn't a huge change, but it should help us towards cleaning up a more awkward aspect of our voice handling. Improvements to voice handling more generally will help us to pave the way for features that rely heavily on voice management, such as audio scheduling. This in turn will allow us to address issues in existing features such as audio queues and sync groups, which can only be used in limited ways and are mostly isolated from the other parts of the audio engine.

[toby-yoyo]

Various things have changed due to this task:

audio_channel_num

• This function will respect the number passed into it.
• An internal lower limit of voices has now been removed, so users can now pass 0 to the function to disable audio playback.
• Channels used to play sync groups are now part of the pool that is resized by this function.
• Note that audio from video playback will still not be affected by this function on most platforms.

audio_debug

• Two new source states will be displayed in the audio debug window:
  • 'Waiting' sources are preparing to play and will begin playback once some internal conditions have been satisfied. These are coloured in magenta.
  • 'Fading' sources will stop after the next mix occurs on the audio thread. These are coloured in cyan.
  • Note that both of these states are transient and will probably only be visible for some time in the order of tens of milliseconds.
• Individual sounds played in sync groups will be visible in this window (see below).
• The numProcessed column has been removed as it will only ever display 0 due to changes to when audio buffers are (un)queued.
• A syncSource column has been added which shows which source each source is synchronised to. This will display -1 for sources that are not syncing and for sources that are being synced to by others.

Audio sync groups

• Sounds in sync groups will now each have their own source. Previously, they were premixed and routed to a single source.
• Sources assigned to sounds in sync groups will no longer be separate from the main voice pool, and so they will be affected by the same rules and behaviours that regular sounds are (voice stealing, audio_stop_all etc.). Note that sync group sounds are currently played with the maximum possible priority.
• Sounds in sync groups no longer need to be exclusively compressed - a mixture of compressed and uncompressed sounds can be played together in a sync group.
• Sync groups will now be supported on the GX.games target.
• Sync group functions taking a sync group index can now print 'bad index' warnings.
• audio_sync_group_debug has been stubbed. Use audio_debug to debug sounds played together in sync groups.

[toby-yoyo]

Most of this task is not user-facing. It basically reworks the systems with which we handle compressed audio and sounds that are streamed from a buffer queue rather than a single static buffer. The user-facing side of this is detailed in the comment above.

Used this project to attempt to test the changes: https://drive.google.com/file/d/1LR-OnaADffs5y6qA9Fu01W0IOkhF3wS_/view?usp=sharing

The old decoding threads would have in the past fallen over when running many of these tests. A number of those issues were patched up for 2024.8 but some will still cause problems in that version too.

The tests don't specifically test features and are not representative of anything like normal GM usage, but they try to stress the audio engine in various areas associated with this task in order to find crashes, data races and similar issues.

[gurpreetsinghmatharoo]

Documentation updated as per Toby's comment

[alicemoretti]

On Beta IDE v2024.1100.0.626 Runtime v2024.1100.0.652, the project attached crashes on HTML5.

Reopening the issue. Thank you.

[toby-yoyo]

Apologies, I should have mentioned that this is a C++ runner-only change and isn't applicable to the HTML5 runner as they have two entirely different audio systems. The project wasn't really written in HTML5 in mind.

That being said, the project did actually expose a few bugs on HTML5, which I've raised the following issues for:

• In-Game: [HTML5] audio_create_stream will crash if the initial request for the stream file fails #7804
• In-Game: [HTML5] Calling audio_sound_loop_end() on a decoding streamed sound will cause a crash #7805
• Building Projects: [HTML5] Streamed sound in Included Files is not exported in the attached project #7806

I've also edited the project to be a bit more HTML5-friendly, which can be accessed from my comment above. With some quick-fixes in for #7804 and #7805 on my branch, the project runs the whole way through without crashing, though it does complain about #7806 which we should fix too.

Re-closing this one as the HTML5 issues are unrelated to the task and have now been logged separately. Thank you for raising that.

[scott-dunbar]

Closing issue - verified in IDE v2024.1100.0.631 Runtime v2024.1100.0.656

As per comments above, not implemented in HTML5, but crashes have been fixed