Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Freeze frames with hardware decoding #2566

Closed
avelad opened this issue May 12, 2020 · 21 comments
Closed

Freeze frames with hardware decoding #2566

avelad opened this issue May 12, 2020 · 21 comments
Labels
status: archived Archived and locked; will not be updated type: external An issue with an external dependency; not our issue; sometimes kept open for tracking

Comments

@avelad
Copy link
Member

avelad commented May 12, 2020

Have you read the FAQ and checked for duplicate open issues? Yes

What version of Shaka Player are you using? master as of 2020-05-12

Can you reproduce the issue with our latest release version? No

Can you reproduce the issue with the latest code from master? Yes

Are you using the demo app or your own custom app? Both

If custom app, can you reproduce the issue using our demo app? Yes

What browser and OS are you using? Chrome 81 macOS 10.15.4

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

What are the manifest and license server URIs?
I have sent an email privately.

What did you do?
Load a Live DASH live stream

What did you expect to happen?
The playback is smooth

What actually happened?
After loading the frame freezes but time continues running. This in version 2.5.11 does not happen.

@avelad avelad changed the title Frame freeze with the latest changes in the master Freeze frames with latest master changes May 12, 2020
@theodab theodab added type: bug Something isn't working correctly and removed needs triage labels May 12, 2020
@theodab theodab self-assigned this May 12, 2020
@theodab
Copy link
Contributor

theodab commented May 12, 2020

Reproduced. Oddly, when you pause and resume the stream, it works fine from that point onwards.
I'll look into it. Thanks for the report!

@shaka-bot shaka-bot added this to the v2.6 milestone May 12, 2020
@theodab
Copy link
Contributor

theodab commented May 13, 2020

Looking at the media internals when this happens, it looks like there's a DEMUXER_UNDERFLOW, then a BUFFERING_HAVE_ENOUGH. When the video unfreezes, there is a DECODER_UNDERFLOW followed by a BUFFERING_HAVE_ENOUGH.

Oddly enough, since Joey updated to the newest master, this seems to be happening a lot less frequently. I did a git bisect which suggests that ca81d55 is the root cause of that. I'm not sure why; the asset you provided does not have any text tracks. Looking at media internals again, the video element is still having periodic DEMUXER_UNDERFLOW events, but they do not lead to freezes.

I'll try to figure out which CL this started with, tomorrow. At first I thought it was e018aaa, but then I got it to reproduce in an earlier revision. In some versions, this appears to happen a lot less often. Or maybe it has to do with what part of the livestream is playing, not sure yet.

@joeyparrish
Copy link
Member

I have sometimes found that a git bisect can sometimes be misleading if the behavior you are testing turns out to be non-deterministic. If you do it again and get different results, you may want to try repro several times in a row at each step of the bisect.

Looking at the media internals when this happens, it looks like there's a DEMUXER_UNDERFLOW, then a BUFFERING_HAVE_ENOUGH. When the video unfreezes, there is a DECODER_UNDERFLOW followed by a BUFFERING_HAVE_ENOUGH.

Is that a typo, or is it the same sequence of events both when it freezes and unfreezes?

@theodab
Copy link
Contributor

theodab commented May 13, 2020

When it freezes, the demuxer underflows. When it unfreezes, the decoder underflows. In both cases, the respective underflow is followed by a BUFFERING_HAVE_ENOUGH event.

@avelad
Copy link
Member Author

avelad commented May 14, 2020

@theodab Do you have any update?

@theodab
Copy link
Contributor

theodab commented May 15, 2020

I've noticed that, sometimes, it seems to freeze only briefly. To try to figure out if this is a result of the period flattening changes, since that is one of the biggest code changes since v2.5.11, I tried running a number of times both with current master as-is, and with current master with some parts of the period-flattening code disabled. The results:

With current master:
Trial 1: Video froze, resumed after about 10 minutes.
Trial 2: Video froze, resumed after about 4 minutes.
Trial 3: Video froze. I let it sit for 15 minutes, and it didn't resume.
Trial 4: Video froze, resumed after less than a minute.

With current master, modified to disable parts of the period-flattening code:
Trial 1: Video froze, resumed after a few seconds.
Trial 2: Video froze, resumed after about a minute.
Trial 3: Video froze. I let it sit for 10 minutes, and it didn't resume.
Trial 4: Video froze, resumed after 3 minutes.

I could run more trials in the future, but for now that's looking like a no.

I also tried running the stream in Firefox and Safari. Firefox played for half an hour with no issues. Safari repeatedly got stuck buffering, but it was actually buffering rather than getting stuck decoding a video frame, so that's probably an unrelated issue.

So it is looking like this might be a problem with the decoding pipeline in Chrome, rather than Shaka Player. The audio keeps playing while the video is frozen, and the messages in media internals both suggest that the video pipeline might be getting stuck. If you don't have any objections, I can show the stream to a Chrome person, to see if they know what could be happening under the hood.

@avelad
Copy link
Member Author

avelad commented May 15, 2020

Yes, you can show the video and share the stream privately with someone from Chrome, you have my permission. Thanks for the research!

@avelad
Copy link
Member Author

avelad commented May 19, 2020

@theodab Do you have any feedback from the Chrome team?

@theodab
Copy link
Contributor

theodab commented May 19, 2020

They're looking into this now. They might open a crbug for this. If so, the plan is to create repro steps that do not require your content (both for reasons of privacy, and also not having to rely on a livestream for repro).

@avelad
Copy link
Member Author

avelad commented May 28, 2020

@theodab do you have any update? maybe a crbug link?

@theodab
Copy link
Contributor

theodab commented May 28, 2020

No news yet. The Chrome team is still trying to isolate a consistent repro for this. I haven't heard anything since last week.

@joeyparrish
Copy link
Member

@avelad, the Chrome team has had no luck in understanding the problem because of the inconsistency in reproducing it. They have some suggestions though:

  1. Try disabling hardware decoding in Chrome. See chrome://flags/#disable-accelerated-video-decode If this affects the problem, we're more likely to have a Chrome bug on our hands.
  2. Something may be wrong with our timestampOffset setting in MediaSource. Try logging timestampOffset in MediaSourceEngine in setStreamProperties, and perhaps also the startTime and endTime in appendBuffer. It could be useful to compare this to the same values in v2.5 if we broke this as part of period-flattening.
  3. Though git bisect didn't work well, the most likely change related to this would be e24fec4. Please try this commit and the commit right before that. This is my best guess to narrow it down considerably.

Thanks!

@joeyparrish
Copy link
Member

They just emailed us again with one more suggestion:

  1. Try adding an abort() call before each appendBuffer(). If this works, it probably indicates an issue with the encoding of the content. Within MediaSourceEngine, right before this:
await this.enqueueOperation_(
    contentType,
    () => this.append_(contentType, data));

Add this:

await this.enqueueOperation_(
    contentType,
    () => this.abort_(contentType));

Note that this appears several times in the method, in branches dependent on transmuxing, closed captions, etc.

@theodab theodab removed their assignment May 29, 2020
@avelad
Copy link
Member Author

avelad commented May 29, 2020

I tried the 1 option and the stream works as expected.

I'm going to try the rest of the points, but I need a little time.

@avelad
Copy link
Member Author

avelad commented May 29, 2020

  1. Both versions have the same values

  2. It doesn't work

  3. will take a little longer, I think on Monday I could see it in more detail.

My perception is that it is something related to hardware decoding ... It is the only change that has made it work.

@joeyparrish
Copy link
Member

If the content fails with hardware decode, there is probably some encoding parameter that you could adjust to improve this. But I don't know what it is. I'll talk to the Chrome team again.

An alternative to option 4 is that you could check if the segments all begin with a keyframe. They should, for what it's worth. But if they don't start with a keyframe, then the content will need to be fixed.

@joeyparrish
Copy link
Member

Chrome team believes that this is seems to be a content issue, but that they also see a chance to improve Chrome's response, either with rejection or a helpful log message. Would you please file a bug on crbug.com and link to it here? I will CC the right people on that issue and they will try to determine the root cause in your content.

@joeyparrish joeyparrish added the type: external An issue with an external dependency; not our issue; sometimes kept open for tracking label May 29, 2020
@joeyparrish joeyparrish removed this from the v3.0 milestone May 29, 2020
@joeyparrish joeyparrish removed the type: bug Something isn't working correctly label May 29, 2020
@joeyparrish joeyparrish changed the title Freeze frames with latest master changes Freeze frames with hardware decoding May 29, 2020
@joeyparrish
Copy link
Member

Someone from Chrome team filed this bug to investigate further: https://bugs.chromium.org/p/chromium/issues/detail?id=1091969

percytse-harmonic added a commit to harmonicinc-com/shaka-player that referenced this issue Dec 28, 2020
@sandersdan
Copy link

sandersdan commented Feb 5, 2021

I've seen issues like this with the macOS hardware decoder in Chrome caused by mismarked keyframes. The implementation will discard frames until it finds an IDR, and if it never finds one then you get the behavior described here.

This should be logged in the media log, and there may also be messages in chrome://gpu.

There was a change made to treat SEI recovery frames with a recovery interval of 0 as if they were IDRs, first available in 89.0.4363.0. Is the stream using SEI recovery? Does the issue still happen in Chrome Beta/Dev/Canary?

@avelad
Copy link
Member Author

avelad commented Feb 8, 2021

I just checked in Chrome Canary(90.0.4411.0 (Build oficial) canary (x86_64)) and it works well. In Chrome stable it doesn't work.

percytse-harmonic added a commit to harmonicinc-com/shaka-player that referenced this issue Mar 12, 2021
percytse-harmonic added a commit to harmonicinc-com/shaka-player that referenced this issue Jun 15, 2021
@avelad
Copy link
Member Author

avelad commented Jan 14, 2022

I have tried again with Chrome 97 and it no longer happens, since time has passed and the problem in chromium has been resolved, I close this issue. Thanks!

@avelad avelad closed this as completed Jan 14, 2022
@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Mar 15, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: archived Archived and locked; will not be updated type: external An issue with an external dependency; not our issue; sometimes kept open for tracking
Projects
None yet
Development

No branches or pull requests

5 participants