Juttering video and audio on Chrome -- SPS/PPS NAL units

[adamscybot]

Strange issues on my stream. Have sent stream to tvstreambox at gmail dot com.

[adamscybot]

I have narrowed this down to Chrome on OSX only (pepperflash).

[adamscybot]

I can now confirm this was broken in the latest version of chrome and only with its built in flash.

37.0.2062.94 works
37.0.2062.120 (latest, releaed last tuesday) breaks

Both versions work on windows fine, so this is osx specific. However, http://osmfhls.kutu.ru/ does not suffer from the problem, so presumably a solution exists.

[adamscybot]

Interesting information surrounding this release:

Version 37.0.2062.120:
This update includes 4 security fixes
Build from 37.0.2062.120 source code
Pepper Flash player has been updated to 15.0.0.152

Flash has been updated. It is a direct port, but clearly a broken one, at least for osx.

[Wrhector]

The issue appears with streams transcoded with mpeg profile "main" v3.1, "baseline" works

[adamscybot]

@mangui note that our bottom 3 profiles use baseline. But the top 2 are the ones that jutter on Chrome OSX 37.0.2062.120 (phew, a mouthful).

So in summary this is a compatibility issue between Chrome >=37.0.2062.120 for OSX (Other platforms WORK) and its included PepperFlash (15.0.0.152) and flashls and chunks encoded with main v3.1 (other main profile may work, but baseline definitely does).

As mentioned, the other player works -- so presumably there is a way to get it to work.

[phloxic]

Just a quick throw-in that I cannot repro, and that's with top profiles at 4.1. Does it work if you turn Pepperflash off?

[adamscybot]

It works with pepperflash off. I've managed to repro across 3 mac machines on 10.9 and 10.10.

Did you mean 4.1 or 3.1? Perhaps this is even more specific.

[phloxic]

I meant both. But I only tested the flashls plugin for Flowplayer, maybe that makes the difference.

[adamscybot]

I have tested osmf and chromeless. I will get round to testing flowplayer. What version of OSX do you use?

[phloxic]

10.9.4

[mangui]

Hi guys, I can repro similar issue with Chrome 37.0.2062.120 with 15.0.0.152 on Win7/64bits

[mangui]

I have the feeling that the issue only happens when playback switch from one level to another ?
also I can see this message in the logs :

"DEBUG:TS: flushing demux" chromeless
"DEBUG:TS: partial audio PES at end of segment" chromeless
"DEBUG:TS: partial AVC PES at end of segment" chromeless
"DEBUG:TS: parsing complete" chromeless

this means that the last audio and video PES packets of some TS fragment are segmented improperly (accross 2 fragments)
maybe there is something wrong around here ... either in your fragment and/or in flashls logic.
I would need to mirror part of the playlist (a 60s snapshot of all levels) locally.
could you share me a snapshot so that I can investigate what is going on ?
cheers,
Mangui

[phloxic]

hm, I get DEBUG:TS: partial AVC PES at end of segment in other browsers too - not the audio, basically at each segment. I do force keyframes at the beginning of each segment though (if that's relevant).

[mangui]

yes, you will get this debug message accross all browsers as the parsing code is generic. i am just suspecting that Flash 15 might behave differently than Flash 14 in case of a missing/broken audio or video frame.

[adamscybot]

Interesting how this seems to very machine specific -- I couldnt repro on Win 8 64bits. The combination of factors required to make this happen must be very specific.

Does my stream have key frames placed properly at the beginning of each chunk (unsure how to check myself)?

I have tested just one playlist profile and still experience the issue -- so I don't think it is strictly related to quality switching.

I should be able to provide a zipped snapshot of the stream later today.

[mangui]

if you have a snapshot (one or multiple level) with which you can 100% repro, that's fine as well.

[adamscybot]

Hi mangui. I have emailed you a snapshot.

[mangui]

Hi @adamscybot ,
I can repro with your snapshot with Chrome/PepperFlash 15,
I also concateneted all fragments into one, and I can repro as well. always repro at the same position.
it is also reproducible when disabling audio parsing/processing (when doing so, the video freeze a little longer).
while analyzing the PES packets, I am suspecting the freeze might be correlated to PES packet having the following sequence :
DEBUG:AVC: AUD, SPS, PPS, SEI, NDR slices

anyway what seems clear is that it should be related to hw acceleration that has been activated by default on Chrome according to http://helpx.adobe.com/flash-player/release-note/fp_15_air_15_release_notes.html

what needs to be checked here :
currently each time I parse a PES packet containing SPS and PPS, I mark it as containing a keyframe. this was to fix some playback issues with some open GOP playlists.
but it should not be the correct way. keyframe should be determined by parsing slice_type.
the other issue might be related to reinjecting SPS/PPS ?

[mangui]

Hi, plz recheck with flashls/master and tell me if it is better.
this code change fixes the glitches i observed on @adamscybot stream.
Cheers,
Mangui

[adamscybot]

I can confirm this has fixed my stream. Have asked clarification from encoder provider about why SPS/PPS NAL unit appears in strange places.

[oripessach]

The latest change causes serious issue (video corruption) for me. I was seeing the problem that the change was meant to address, by the way.

I'll try to put together a stream that demonstrates the problem.

[mangui]

yes, if SPS/PPS change accross times, you will have video corruption. the code change I did is more a workaround than a real fix I am affraid. please share your stream, i need to think about a real solution.

[mangui]

Hi all,
I would advise you to recheck your streams with flashls/master.
this fix should be more accurate as the previous one as some obvious flaws.
Cheers,
Mangui

[oripessach]

Here's a stream that shows very choppy playback in Chrome, but plays well on Firefox and in VLC:

http://software.envysion.com/testvideo/manifest.m3u8

The encoder I'm using seems to produce new SPS/PPS before every IDR, and it always sends an IDR at the beginning of a fragment. The encoder only produces baseline profile, by the way.

[mattwa1sh]

Yes, we're seeing the same thing. Going to build the latest release of chromeless player and see if it's fixed. Kudos to Mangui for being on top of this.

[mangui]

Hi
i can repro the issue with flashls/master, chrome/win7 and http://software.envysion.com/testvideo/manifest.m3u8
also, could you confirm that jw native also provides hardware acceleration ?

I will try to see how this could be fixed with this stream.

@ALL, any other broken playlists welcomed.

[mangui]

I checked http://software.envysion.com/testvideo/manifest.m3u8 with h264bitstream
I can see that SPS/PPS is changing with every keyframe.

I am reinjecting a new AVCC as expected with updated SPS/PPS... not clear yet why it does not work with hardware acceleration. will digg further.

[mangui]

Hi @mattwa1sh
http://blive-video.bproductions.com/000886-002087-pu5/hls/desktop-playlist.m3u8 is multibitrate, could you tell me with which level/subplaylist you can observe an issue @ 1mn46s ?

[mattwa1sh]

Okay, I see this if I set the level above 4. At lower bitrates I do not see the issue in Chrome, which I believe would correspond with 4628, 2628 and 1928.

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4628000,CODECS="avc1.64001f",RESOLUTION=1920x1080
file-4628k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2628000,CODECS="avc1.4d001f",RESOLUTION=960x540
file-2628k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1928000,CODECS="avc1.4d001f",RESOLUTION=960x540
file-1928k.m3u8

[mangui]

so you mean the issue could be observed with any of the 3 level above at exactly 106s ?

[mattwa1sh]

Yes, I observe the issue at levels 4,5, and 6 by setting them via "set level" in the chromeless player.
Thanks!
Matt

[mangui]

i am not sure to visualize the same thing here, nothing really obvious with chrome/pepperflash 15/Win7
could you try to set max buffer size to 1000 and play from 0 up to 110s, and tell me if you still see your glitch ?

[oripessach]

Should the SPS/PPS not change every time? Since they have to be sent at least once per segment (which is my understanding from reading http://tools.ietf.org/html/draft-pantos-http-live-streaming-13#page-20) you'd get at least one SPS/PPS set for each segment... No?

[mattwa1sh]

I watched from 0 with the max buffer at 1000. The playback is still a tad bit jittery for me, but I didn't see the major tick at 1:46. When I seek back, I see the tick again.

The 1:46 thing is as an example of the kinds of things we've been seeing in lots of our videos during the past two weeks.

Like, in the one below for Tory Burch, there's a replicable tick at :16, when the girl in the yellow jacket turns around, that happens even if I watch from beginning, even with max buffer = 1000. I see it when I set level > 3. It happens every time.

http://blive-video.bproductions.com/000929-002407-9j8/hls/desktop-playlist.m3u8

I taped what I'm seeing on my phone at level 4.:

https://www.youtube.com/watch?v=H1zi3sSFW2E

M

[mattwa1sh]

Actually, some kind of tick happens there. The severity is sometimes less.

[mangui]

i think there are two issues with your stream :

• dropped frames when buffering/parsing is in progress
• video glitch (only happening with hw video decoding)

for the dropped frames, this seems to happen because too many times is spent in ENTER_FRAME event for TS parsing/and tag injection. we might have to review the max allowed time here.

regarding video glitch, it should be the same issue than for the other streams. I am still not clear about the root cause. I rechecked the logic and I dont see anything wrong... if anybody has a contact @ Adobe it might help, I have a couple of questions to ask ...
Cheers,
Mangui

[mattwa1sh]

Not sure if this is a dumb question, but can we disable hardware acceleration all together? The video plays fine in Safari and FF after all.

[mangui]

Hi @mattwa1sh
i added one additional log to understand how FP is handling the video decoding.
with Chrome/Win7/PepperFlash, I got the following logs displayed

[16:44:15] switching state to PLAYING (index):212
[16:44:15] onFragmentLoaded(): bandwidth:218477946 level:6 width:640 (index):212
INFO:Video decoding:software VM4136:1
[16:44:15] onFragmentLoaded(): bandwidth:184496176 level:6 width:640 (index):212
[16:44:15] onVideoSize(), 1920x1080 (index):212
[16:44:15] onVideoSize(),resize stage to 1280x720 (index):212
[16:44:16] onFragmentLoaded(): bandwidth:160774815 level:6 width:1280 (index):212
[16:44:16] onFragmentLoaded(): bandwidth:235748042 level:6 width:1280 (index):212
[16:44:16] onFragmentLoaded(): bandwidth:242541430 level:6 width:1280 (index):212
[16:44:16] onFragmentLoaded(): bandwidth:217951697 level:6 width:1280 (index):212
[16:44:25] onFragmentPlaying(): metrics:6 sn:1 cc:0 tags:#EXTINF:9.9766, (index):212
[16:44:25] onFragmentLoaded(): bandwidth:288587191 level:6 width:1280 (index):212
INFO:Video decoding:accelerated VM4298:1
INFO:Video decoding:software
...
...

AND the glitch happens exactly when Stage video switches from software decoding to accelerated decoding ... I am not clear exactly why this is happening in the middle of the playback as there is no explicit trigger to switch from software to accelerated. it should be pure FP internal decision.

[mangui]

it is possible to disable hw acceleration, just replace this line
https://github.com/mangui/flashls/blob/master/src/org/mangui/chromeless/ChromelessPlayer.as#L413
by
var available : Boolean = false;

[mattwa1sh]

That is odd. I will remove the acceleration (for now), re-compile and see what that does. Stand by.

[mattwa1sh]

Effing eff. That seems to have fixed the problem in Chrome:

http://blive.bproductions.com/public/examples/chromeless/

Also, I didn't set the max buffer. It works okay with just the default. And I watched on level 5, which formerly had a lot of problems.

It plays fine in Safari too.

Matt

[oripessach]

The same change seems to work for me, too.

[mattwa1sh]

Well, the electrician severed my broadband connection today and I am tethering to my phone instead. I can sometimes still see the problem at level 6 in Chrome with hardware acceleration off. But with this setup I haven't ever seen the problem at level 5. So the issue is not fully resolved.

But that raises the question. How can I possibly get level 6 over 4G? I only have 3 bars on my phone here. Maybe a more conservative bandwidth to level calculation should be used. I will do some more testing tomorrow when I have a better internet connection.

Matt

[mangui]

I checked chrome://gpu/ on my side and I can see that my GPU is blacklisted, so hardware decoding is disabled. but still flash is trying to use it intermittently ...
I then disabled this setting : chrome://flags/#ignore-gpu-blacklist to force hw acceleration,
when it is forced, I can see in the logs that hardware video decoding is always enabled, and playback is smooth.
so this seems to confirm that this video glitch is introduced by FP intermittently switching video decoding from hw to sw.

[mattwa1sh]

I'm not sure about that. I made the followinf change ~L440 in ChromePlayer.as:

---

    protected function _onStageVideoState(event : StageVideoAvailabilityEvent) : void {
        var available : Boolean = (event.availability == StageVideoAvailability.AVAILABLE);
        available = false; //hack to test chrome hardware acceleration

---

And I can see the tick depending on how I seek, etc. This is a complex issue that is brought to light by, rather than caused by, the hardware acceleration. It's not a 1:1 relationship. I need to really dig in and test more, but this is still out there.

[oripessach]

On my machine, hardware decoding is enabled in chrome (the GPU isn't blacklisted), but the problem is still present unless I disable hardware acceleration in flashls.

[mattwa1sh]

I think we can say the problem is greatly attenuated or eliminated by disabling hardware acceleration,

[mangui]

I analyzed the glitch with Adobe Scout. when Flash is switching from software to hardware video decoding, we could see some strange activities around NetStream and Video Decoding.

also we could see memory allocated. a debug version of pepperflash would be needed to investigate what is going there.

[mattwa1sh]

That seems right. I built "allowHardwareAcceleration" as a flashvar which I check against the user agent. I'm allowing for all browsers but Chrome.

[mattwa1sh]

Yeah, I will get this documented and put it up. Things have been hectic
here and I'd definitely like to put up something complete, so I need to
make a good example.

On Wed, Oct 29, 2014 at 11:03 AM, Adam Thomas [email protected]
wrote:

@mattwa1sh https://github.com/mattwa1sh Would you be willing to share
your changes/compiled chromeless swf?

—
Reply to this email directly or view it on GitHub
#66 (comment).

[mangui]

Hi, with hls_usehardwarevideodecoder=no, I cannot repro the issue anymore, could you confirm on your side ?
Cheers,
Mangui

[mangui]

@mattwa1sh is it still happening on your side with latest flashls/dev ?

[mattwa1sh]

Yes with hardware acceleration off this was not a problem. I do not have latest flashhls.... I really ought to do a pull request for my changes.