[ViacomCBS] LG Ad Period Stalls

[dsparacio]

Have you read the FAQ and checked for duplicate open issues?
Yes

What version of Shaka Player are you using?
2.5.14, 2.5.16, 3.0.4 - Happens in all three versions

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Yes

Are you using the demo app or your own custom app?
Demo app produces same issue as our player

If custom app, can you reproduce the issue using our demo app?
Yes

What browser and OS are you using?
WebOS

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

In the 2.5.x version this was the only one that had issue.
Device: LG 2019
Model: 32LM620BPUA
WebOS: 04.72.10

In 3.x version
Platform: LG
Device: 49UM7300AUE
WebOS: 04.80.10

Device: LG 2019
Model: 32LM620BPUA
WebOS: 04.72.10

What are the manifest and license server URIs?
Will send content and script to get DAI version and DRM info etc.

What did you do?
Transition from preroll to content
Transition from content to 1st midroll
Play all ads from last midroll & Transition to content

What did you expect to happen?
It would play the Ad periods and Content periods without stalling and requiring a seek to march on.

What actually happened?

It stalls at that boundary (only on some LGTV and some content).

Temp workaround 90% effective, stallSkip was set to 1 second. This work okay for most LG tvs but not the two mentioned above. Got worse in 3.x

Perhaps a DAI Ad encode or conditioning/insert issue?
I say this because, our content without "DAI influence" plays well across periods on LG. I first thought of DRM init issue like we saw with Samsung, but in this case once stalled, I can seek 1 second forward to get it unstuck, which I could not do with the DRM issue.

As mentioned we found stallSkip setting but I also found this warning in the docs:

stallSkip: number - The number of seconds that the player will skip forward when a stall has been detected. If 0, the player will pause and immediately play instead of seeking. A value of 0 is recommended and provided as default on TV platforms (WebOS, Tizen, Chromecast, etc).

Since StallSkip does not solve for all LGs and this warning we figured we should start the ticket and see what your team thinks.

The fact I can seek to the other side of that "gap" and get playback to continue, leads me to believe encoding is contributing something to this issue. We think since it is inconsistent with our content, that some ad encodings and some of our content encoding is not meshing well on with this decoder.

[joeyparrish]

Since we don't have an LG TV, we won't be able to debug this directly. But we will try to suggest things for you to try and logs you could collect to help find out what's going.

Since you mentioned seeking and gaps, have you tried enabling the streaming.jumpLargeGaps config?
https://shaka-player-demo.appspot.com/docs/api/shaka.extern.html#.StreamingConfiguration

If that works, it might indicate an encoding problem, but that config is a fairly benign thing to have enabled. If it works, I would recommend you use that as a solution.

[dsparacio]

Yes sorry forgot to mention we tried jumpLargeGaps and smallGapLimit before we found stallSkip. Neither gap jumping setting did anything at any level. I have not tried to set the Gap settings in addition to the stallSkip. Wonder if that would help in the cases were stallSkip is not working. I will test this and get back to you.

Are your docs still accurate, should we be worried about trying to use the stallSkip on tvs. What is the reason for that?

[dsparacio]

Tested again with stall and jump settings. no change. Logs are very minal. One thing that popped out was this:
streaming_engine.js:1435 (audio:1) need Period 1 presentationTime=0.458 timeNeeded=30.016 currentPeriodIndex=0
streaming_engine.js:1435 (video:2) need Period 1 presentationTime=6.006 timeNeeded=30.029 currentPeriodIndex=0

Mean anything to you? It is getting stuck at this period bounds...

[dsparacio]

About to dig into this one end of week or beginning of next week. Just wanted to add comment so not closed by bot. Still on Radar but XBOX is first.

[joeyparrish]

@dsparacio, is this still an issue with the recent v3.0.8 release? (We still don't have an LG TV set up.)

[dsparacio]

It is still an issue, not our top priority, so we have not tested on latest. Let me get back to you after we retest soon.

[joeyparrish]

Sounds good. Thanks!

[dsparacio]

@joeyparrish Yes this is still an issue and so is this
#2998 that has been locked and archived. I am wondering can we get this one open again?

[TheModMaker]

As mentioned in that issue, that is a duplicate of #2957. Does your issue here only happen when playing text streams? If you comment out this line, does it work?

https://github.com/google/shaka-player/blob/6db55e0141574a80872b1ab20f69a3720a2f0c91/lib/media/streaming_engine.js#L1036

[dsparacio]

@TheModMaker Sure, but I mentioned in my ticket that I dug in and found that, and it does solve if I comment that out but was told that is a hack and not the real solution. It will allow for run away text scheduling. The issue is really is around the fact that flatten period should say this period has no text and that logic so not be forced to eval, right? I am just trying to track our open issue so Ill follow #2957

[joeyparrish]

@TheModMaker is assigned to #2957 and we have a goal to fix that before v3.1 launches.

@dsparacio, thanks for confirming that this is still an issue. Does it occur on any platform other than LG? We still don't have such a device in our lab.

Also, I searched my email and couldn't find any manifests or instructions associated with this issue number. Did I overlook it, or does the email not mention this issue number?

[joeyparrish]

Oh, and I just realized that this looks very similar in symptoms to the issue we fixed for Xbox & Tizen. So here's a really cheap thing you can try on WebOS. Edit this method in lib/util/platform.js:

https://github.com/google/shaka-player/blob/6db55e0141574a80872b1ab20f69a3720a2f0c91/lib/util/platform.js#L296-L308

And add || Platform.isWebOS() to that condition. That will enable the same multi-period-DRM workaround for WebOS that we built and tested on Xbox.

If that fixes it, we can get that committed and in v3.0.9 pretty quickly.

[MilosRasic]

We had exactly the same issue on all LG models with the same kind of content and it turned out to be caused by gaps in the manifest which are in turn caused by b-frames in both ads and encrypted content. LG is far more sensitive to these than other devices. We solved this by encoding ads without b-frames for LG in order to minimize the number of gaps and applying a workaround to detect stalls and unstall playback by moving currentTime by a fraction of a second, 0.8 to be exact.

Should be in production soon so that we can see whether the hardcoded value of 0.8 does the trick on all the models in the wild, but I'd definitely love to see a more universal solution in Shaka Player.

Not sure how our solution differs from Shaka's stallSkip option. Maybe @bcupac can shed some light on that.

[dsparacio]

@joeyparrish thanks for that update let me try the addition of webos check first and get back to you on content if needed.

[dsparacio]

Does it occur on any platform other than LG? We still don't have such a device in our lab.

Not that we have reports on, this particular one seems to be isolated to LG and DAI content period transitions

[joeyparrish]

@MilosRasic, I have been thinking about our gap-jumping/stall-detection for a while, and I am starting to think it needs to be burned to the ground and redone completely. It may have been built on bad assumptions, and it was definitely built to compensate for browser behavior from many years ago. Desktops have moved on, and it's CE devices that really benefit most from manipulating the timeline and seeking automatically.

[dsparacio]

@MilosRasic so mostly same for us.. not all models but enough.. then we implemented stallSkip but shaka docs advises not to for these platforms. We found it did work a bit, but did not solve it for all LG models.

We then took stallSkip out in fear that it would cause bigger issues in prod. Turns out this is reported and widespread so we may just turn stallSkip back on but still not sure yet.

@joeyparrish Let me catch up on this thread and get you more info and content and test more in next couple days.

[bcupac]

Yes, we solved most of our SSAI gap problems by having ads with no b-frames, as b-frames somehow cause these gaps. Our content still has b-frames so there are still gaps where the content ends and ads begin.
Until all of that is resolved and until we have ads and content with b-frames and no gaps, we use seek to unstall video.

Shaka's stall recovery for most platforms uses seek to unstall video, but for LG it uses only play/pause, and play/pause does not unstall video on some of our test devices for some reason.
The explanation for not using seek on LG is that it has a long hardware pipeline, which I think means that it is slow to seek, so play/pause was considered a much faster unstall mitigation mechanism at the time.

[dsparacio]

Great info @bcupac thanks for sharing

[dsparacio]

@joeyparrish I am picking back this up, will supply result asap. May be a few days to get the person with the LG that reported it to test. They are out this week. Just a heads up.

[joeyparrish]

No worries. Thanks!

[dsparacio]

@joeyparrish Ok tested the DRM fix on LG and that is not the issue. For this, it GAP related and I do see the pause/ play log line vs the seeking to try to recover the GAP.

I will try some of GAP settings and get back to you.

[dsparacio]

@joeyparrish we are going to try some encoding experiments on our side to see if any mixture works but since we can not control the ad inventory encodes, not sure it will help beside uncovering why we are freezing at best.

I wonder if a better approach to this is on player side to update the stall detection and recover logic in general for these smart TV? Seems like the only way to cover the fragmentation between model years and manufactures, is brute force at this time.

This issue and the "period without text track causes stream to stall duplicate I filed" are our two biggest issues, overall most of or devices are working well.

[joeyparrish]

Hi @dsparacio,

We just put out v3.0.9 with this fix: 01ce0f4 (#3191)

It seems possible that this issue could also affect LG. It's not platform-specific, but the manifestation of it is very inconsistent, and it's always possible that the LG platform makes a certain non-deterministic issue happen more frequently. (Especially since this one involved specific seek times, and the precision of video.currentTime is platform-specific.)

Could you please try with v3.0.9 and let us know if anything is improved? Thanks!

[rroslaniec]

We tested it on LG webOS 4.0 - still video stalls. Right now we will try #3008 (comment) solution.

[dsparacio]

@joeyparrish Just got back from PTO and wll test this on our LG again. Get back to you soon and will check out the #3008 item as well.

[MilosRasic]

We've found out that Vewd devices (Hisense Vidaa, Tivo STB, etc), former Opera for Devices, are also affected by this problem.

#3008 hack helps but if a fix is implemented in Shaka, it shouldn't be limited to webOS only.

[dsparacio]

We are seeing great improve on our LG and Samsung device in 3.0.10. Regardless if directly or indirectly resolved by the work, we are in a better place and very thankful for your help on this.

[joeyparrish]

Okay, great! It sounds like this is solved by v3.0.9/v3.0.10 and #3191. Thanks for the update!

I'll go ahead and close this as solved, but if I'm mistaken, just put @shaka-bot reopen in a comment. Thanks!