slow or distorted sound playing

[ousia]

@Herschel,

this comes from #183 & #203:

1. Sound is played too slow in https://www.free-culture.tk/lfc-presentation.swf (downloaded and played with Linux desktop program [since now standard Firefox extension or web, [extension] SWF file isn’t displayed at all  #98]).

   I already reported this in [AS2] distorted sound playing lightspark/lightspark#381. The root of the problem was that (as @dbluelle kindly explained to me):

   Just FYI: your MP3 has a sampling rate of 16 kHz, but the DefineSoundTag only has flags for 5.5, 11, 22 or 44 kHz.

   This is the reason why this is played too slow. The player from Adobe has no problems with the sound in that SWF file.

2. Sound in https://randomfoo.net/oscon/2002/lessig/free_culture.swf is played distorted. Playing speed is fine and my poor command of the English language doesn’t allow me a more accurate description (I’m afraid you have to listen to it).

   The sound is MP3, mono, 22kHz and 32kbps.

   Pressing any of the buttons (code in sound playing in timeline #203) removes distortion. I wonder whether this is related to Sound.stop() or Sound.play(time).

Many thanks for your excellent work and your help.

[ousia]

@Herschel,

I can’t code (it’s all Greek to me), but I wonder whether the following snippet could fix the issue described in my previous first issue:

        let sample_rate = match (flags & 0b11_00) >> 2 {
            0 => 5512,
            1 => 8000,
            2 => 11025,
            2 => 12000,
            3 => 16000,
            4 => 22050,
            5 => 24000,
            6 => 44100,
            _ => unreachable!(),
        };

For obvious reasons (I don’t even know whether this makes sense at all), I’m not providing a pull request.

I have only copied a snippet from read.rs and I added the missing sample rates.

Just in case it might help 😅.

[Herschel]

Thanks for the report @ousia, for the first issue, I can confirm the issues is the embedded MP3 is of a different bitrate than is specified by the SWF tag (another issue caused by the fact this SWF is generated slightly-incorrectly by 3rd party software.) We'll have to read the sample rate from the MP3 data directly instead of trusting the bitrate in the SWF tag.

For the second issue, it seems like the sound is being played twice on top of itself. Once from the event StartSound tag, another from ActionScript. The StartSound tag has the "no multiple" flag set, so this should prevent it from playing twice, but it's not right now (maybe it's happening out-of-order).

This is a good test case, thank you!

[Coder-256]

I would love to take a shot at fixing this!

[ousia]

@Coder-256, how about providing a pull request?

[Coder-256]

@ousia It's in the works! It's a bit of a large change though. I'm working to refactor the decoders to emit frames instead of individual samples. This will make both decoding and playing more efficient (especially for WebAudio) and will also help prepare to continue implementing streaming. However, this means that I need to essentially rewrite the core audio decoders and the web/desktop audio backends. I'm going to open a draft PR soon, just waiting until I'm confident about the new design for the decoder interface.

[ousia]

@Coder-256, these are great news: huge thanks for that!

[Coder-256]

Ok there is still a LOT of work to be done but I pushed a branch with some of the progress. This ended up being a far more enormous change than I had imagined, and I still have more WIP local changes... plus I need to update the players but it's getting there.

[ousia]

@Coder-256, huge thanks for the improvements.

I’m looking forward to using your improvements.

[Coder-256]

@Herschel There are a few hurdles in the way of VBR support. Sadly I think the web player might have to move away from WebAudio's decodeAudioData, in order to properly support streaming sounds and seeking. I think the best way to move forward is to actually process all audio data with a script processor that delegates to ruffle_core. My plan is to refactor a lot of the similar/duplicate audio handling code into the core, actually implement audio decoding, envelopes, etc. inside the core, and essentially just expose some sort of simple signal/stream interface. The audio backends would just provide a way to play audio, and not deal with decoding audio or the SWF. What do you think?

[Herschel]

@Coder-256 Thanks for your work! That's what I had in mind, as well. Most of the decoders are already in core, so it makes sense to move some of the mixing and envelope code in there, as well. Similarly there was already code in web/audio to use ScriptProcessor, just mostly deactivated and unused currently.

I do think it'd be nice to have two separate web audio backends, one that uses ScriptProcessor and one that uses the current WebAudii API, just for comparison/education/benchmarking. But that's more of a separate change where I want to reorganize all the backends in general to allow multiple choices per platform.

[ousia]

@Coder-256, are there any news on this?

[Coder-256]

I have been a bit preoccupied, but I will rebase and start working on it again.

[ousia]

Is there any news on this particular issue?

The way this kind of presentations are generated, Ruffle cannot deal with any of them.

I’m afraid this is caused by framerate set by lame.

Many thanks for your help.

[Herschel]

No progress here yet unfortunately, but I should tackle it soon when I tackle the remaining Sound APIs (volume, panning, streaming from a URL, etc.)

[ousia]

Many thanks for your reply, @Herschel.

These are really good news. I don’t use Flash for games and I need that (besides Sound.stop(), Sound.position() and Sound.duration()) even to check for other bugs in the AS2 I normally have to deal with.

[ousia]

Just in case it isn’t clear, only the desktop version is affected by the issue of different sampling rate than the ones allowed (in this case, 16 kHz instead of 5.5, 11, 22 or 44 kHz).

[Herschel]

@ousia Thanks for your reports and support. The slow sound should finally work correctly.

The distorted/doubled playing in the linked free-culutre.swf is a problem in the original Flash file that causes the audio to play twice. This doubled playing occurs in the official Flash Player as well, so Ruffle is accurate here.

[ousia]

Many thanks for fixing the issue, @Herschel.