web: Decode and mix audio via Rust

[Herschel]

• Add a new WebAudioMixerBackend that uses the AudioMixer on web, outputting the audio stream via a pair of AudioBufferSourceNodes.
• Use symphonia for MP3 decoding on web.

This gives consistent audio between the web and desktop builds. On web, it should significantly improve loading performance and memory usage, as the AudioMixer can decode the audio on demand, as opposed to decoding completely on load using decodeAudioData. This also allows us to simplify some of the preloading code, as the API was contorting itself to deal with SoundStreamBlock data within the limitations of WebAudio.

Performance may be more intensive during playback as audio is decoded on the fly and not leveraging the browser's native audio capabilities. Underruns may also be an issue, for example, if some AS code does a busy loop, then the audio may drop out.

We should eventually move towards using AudioWorklet instead, and only use AudioBufferSourceNode as a fallback. We can defer that part to a future PR, as a full AudioWorklet implementation will be quite involved and may require us to deal with shared memory.

Thx to @torokati44 and @Toad06 for the assistance!

[ousia]

@Herschel, many thanks for this new implementation

It would be great to have Sound.position() implemented (this is a long awaited feature [not only by myself]).

Just in case it is relevant, at least in MP3 sampling rate should be read from file itself (since Flash already allows unspecified values such as 16 kHz [as reported in #335]).

Many thanks for your code.

[ousia]

If Sound.position() is implemented, this would fix issues #340, #1898 and #2235.

Just in case it might help.

[Toad06]

I've done some testings using the web extension (Firefox). Not sure that's very useful because this PR is still a draft, but here are my observations.

Ultimate Flash Sonic

• Sound effects are sometimes played with a little delay, especially when you collect multiple rings. I suppose the future implementation of AudioWorklet will prevent this issue from happening?
• The "spin dash" sound effect is not played as it should be.

Sonic Shorts Volume

• The issue described in this comment (#1673) is fixed.
• A panic can happen (index out of bounds: the len is 0 but the index is 0). To reproduce, pick "Scene Select", "Next" and click the first image at the top of the file.

Steppenwolf 1-1

• When entering the ship's hold, a sound effect was ending too early; this issue is fixed.
• A sound effect is not played as it should be. To reproduce (I recommend lowering the volume if you use earphones), go up (hold Shift to run), and press space to use the switch close to the small boat.
• Instances of a same sound should be limited in number #1388 still happens but sound instances are not immediately played after the previous ones (audio samples:
  Flash + current Ruffle builds and this PR).

Garfield Coop Catch

• Music no longer loops.

[sombraguerrero]

#3663 would seem to tie into this.
For Onda Educa #4333 is most likely an expression of this (frameskip on "children" SWFs)

[ousia]

@Toad06, could you check whether https://www.free-culture.tk/lfc-presentation.swf is played with distorsion or not?

Many thanks for your help.

[sombraguerrero]

@Toad06, could you check whether https://www.free-culture.tk/lfc-presentation.swf is played with distorsion or not?

Many thanks for your help.

@ousia I don't hear any distortion other than distortion I would expect from the source audio being a fairly low bitrate.

[ousia]

@sombraguerrero, many thanks for your help.

Are you playing the SWF file with the desktop program? This is what plays distorted sound.

[Herschel]

Fixed the panic in the Sonic Shorts cartoon. Weird dash sound in Ultimate Flash Sonic seems to be a decoding issue in symphona, filed an issue. Wasn't able to repro the Garfield Coop Catch issue, it seems to loop for me (is it the in-game music?)

[ousia]

The issue reported in Symphonia has been fixed (no idea when the new version will be released).

Just in case it helps.

[ousia]

Symphonia, release 0.3.0 contains the fix for pdeljanov/Symphonia#28 (reported by @Herschel).

Just in case it might help.

[hatal175]

Waiting for this merge for a loadSound pull request (some issues with asynchronous decode_audio_data).

[kmeisthax]

So, I wound up making a few small changes in #5059 that will conflict with this PR if either of them are merged. Specifically, I added get_sound_size and get_sound_format, which lets you get the compressed data size and sound format data for a previously registered sound. Every (real) backend stores it, so it seemed a reasonable thing to do; and it should be fairly easy to migrate over once either of them land. I just wanted to make a note of it here.

Also, I noticed you added mixins for an audio backend base. I think it might be useful to have NullAudioBackend also use that. I already had to make it store some information already so AS3 sound tests would work, and there's future features like extract or leftPeak/rightPeak that I'm not implementing, but won't be testable without NullAudioBackend doing audio decoding like the other backends.

[zumbasana]

Hi Ruffle dev team, @kmeisthax does your note mean that this Pull Request wasn't implemented? We've downloaded most recent Ruffle version, but our sound.position issue still stopping play (see my Issue #5019 ).... Thank you for any advice!

[sombraguerrero]

Hi Ruffle dev team, @kmeisthax does your note mean that this Pull Request wasn't implemented? We've downloaded most recent Ruffle version, but our sound.position issue still stopping play (see my Issue #5019 ).... Thank you for any advice!

When a PR is still in the draft state, it is not part of the main code branch (that's what 'merging' generally means). The only way you would be able to interact with these changes is if you happened to fork the branches that the rest of the developers are working on.

[ousia]

@zumbasana, [WIP] in title stands for “work in progress”. This is still a draft.

As far as I know, there is no commit (no code yet) that deals with Sound.position() in AS2/3 in this pull request.

The lack of Sound.position() was reported more than 19 months ago.

I’m afraid we have to work until this pull request gets merged (if it contains the implementation of Sound.position()).

I hope this makes the situation clearer and I apologize if this contains explanations obvious to you.

[zumbasana]

Thanks much @sombraguerrero and @ousia , I'm a complete non-coding newbie, so your elucidation appreciated. We've got several AS2 projects using sound.position ..... is it possible to bump up a Ruffle translation of that, and work with you guys to get it resolved? Thank you!

[ousia]

@zumbasana, according to what @kmeisthax wrote in #5059 (comment), Sound.position() seems hard to implement, since current code has no way to know the sound position (my uneducated guess 😅).

I cannot code, but I wonder whether the implementation of Sound.position() would be faster if it were independent from this pull request.

Other than that, @Herschel (as the author of this pull request) could explain better how long it would take to get Sound.position() working and how to help him.

[zumbasana]

Thanks @ousia , and @Herschel might a sound.position solution be worked on and implemented independent of this Pull Request? Thank you for your advice!

[Herschel]

Ok, let's see how this goes!