Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On a VOD SAI period transition the player starts loading segments ~7.5 minutes past current position #3191

Closed
caridley opened this issue Mar 2, 2021 · 20 comments
Assignees
Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone

Comments

@caridley
Copy link
Contributor

caridley commented Mar 2, 2021

*Have you read the FAQ and checked for duplicate open issues?
Yes - I think this issue might be related
#3048 Abr Manager Variants out of sync on multi period live DASH playout

What version of Shaka Player are you using?
3.0.8
The problem does not occur with Shaka 2.5.x versions

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Yes - v2.5.20-master-32-g2d52f921

Are you using the demo app or your own custom app?
Custom app

If custom app, can you reproduce the issue using our demo app?
Will try

What browser and OS are you using?
Chrome 88.0.4324.182 on Mac OS 10.15.7

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

What are the manifest and license server URIs?
If and when you want to reproduce we will provide access to streams

What did you do?

Play a Charter VOD SAI stream with multiple ad break and periods that may have different bandwidth ladders.

What did you expect to happen?
Play back should progress smoothly through all of the main and advertising content.

What actually happened?

When currentTime crossed a period boundary in the stream the player started fetching video segments 7.5 minutes ahead of the currentTime. When the currentTime reached this 7 minute gap the player went into a permanent buffering state.

Complete debug logging can be found in this file -
log.txt

2021-02-25T19:29:12.540Z DEBUG CONSOLE (video:87),timeNeeded=2094.095866666666
2021-02-25T19:29:12.540Z DEBUG CONSOLE (video:87),update_:,presentationTime=2053.575858,bufferedAhead=39.65242999999964
2021-02-25T19:29:12.540Z DEBUG CONSOLE (video:87),fetchAndAppend_:,presentationTime=2053.575858,reference.startTime=2094.0958666666666,reference.endTime=2096.092011111111
2021-02-25T19:29:12.541Z DEBUG CONSOLE (video:87),fetching segment
2021-02-25T19:29:12.541Z DEBUG CONSOLE fetching: reference=,[object Object]
2021-02-25T19:29:12.575Z DEBUG CONSOLE Segment downloaded:,deltaTimeMs=108,numBytes=32141,lastTimeChosenMs=1614281348511,enabled=true
2021-02-25T19:29:12.575Z DEBUG CONSOLE Suggesting Streams...
2021-02-25T19:29:12.576Z DEBUG CONSOLE Calling switch_(), bandwidth=15418 kbps
2021-02-25T19:29:12.576Z DEBUG CONSOLE switch_
2021-02-25T19:29:12.576Z DEBUG CONSOLE switch: switching to Stream (video:88)
2021-02-25T19:29:12.576Z INFO CONSOLE Aborting current segment request.
2021-02-25T19:29:12.577Z DEBUG CONSOLE switch: Stream (audio:12) already active
2021-02-25T19:29:12.578Z INFO SHAL.UNIFIED_MEDIA_PLAYER bitrate adjusting to 5548000 from 4923000
2021-02-25T19:29:12.584Z DEBUG CONSOLE (video:88),updating in 0 seconds
2021-02-25T19:29:12.585Z DEBUG CONSOLE Segment downloaded:,deltaTimeMs=10,numBytes=16839,lastTimeChosenMs=1614281352576,enabled=true
2021-02-25T19:29:12.586Z DEBUG CONSOLE Suggesting Streams...
removed some lines here...
2021-02-25T19:29:12.586Z DEBUG CONSOLE Still within switch interval...
2021-02-25T19:29:12.587Z DEBUG CONSOLE (audio:12),checking buffer length
2021-02-25T19:29:12.587Z DEBUG CONSOLE (audio:12),buffer behind too large:,presentationTime=2053.497674,bufferedBehind=12.086909000000105,bufferBehind=10,overflow=2.086909000000105
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),timeNeeded=2094.095866666666
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),update_:,presentationTime=2053.624484,bufferedAhead=39.603803999999855
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),looking up segment from new stream endTime:,2094.095866666666
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),fetchAndAppend_:,presentationTime=2053.624484,reference.startTime=2545.1189999999997,reference.endTime=2547.1276666666663
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),setting timestamp offset to 2545.1189999999997
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),setting append window start to 2545.019
2021-02-25T19:29:12.589Z DEBUG CONSOLE (video:88),setting append window end to 2575.258
skip some loggin
2021-02-25T19:29:14.033Z DEBUG CONSOLE (video:88),advancing to next segment,[object Object]
2021-02-25T19:29:14.033Z DEBUG CONSOLE (video:88),finished fetch and append,[{"start":2045.617999,"end":2093.228288},{"start":2545.152365,"end":2547.154365}]

@caridley
Copy link
Contributor Author

caridley commented Mar 2, 2021

Here is the manifest player config and some metrics

player-manifest.txt
player-config.txt
player-metrics.txt

@caridley
Copy link
Contributor Author

caridley commented Mar 2, 2021

The problem appears to be associated with transitioning between periods with different bandwidth ladders.


when current time was 2053 which is just after the period transition at 2051 within the ad break 1976 - 2081
we had buffered 2045 - 2093
we then suddenly load 2545 - which is start of the next ad break

ad prior to 2051 (2021 ) is http://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2021021200001806/manifest.mpd - period 18
ad at 2051 is http://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2020111900024306/manifest.mpd - period 19
ad at 2545 is http://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2021020200005406/manifest.mpd - period 22

position 2021
<Period id="18" duration="PT0H0M30.163S">
<BaseURL>https://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2021021200001806/</BaseURL>
<AdaptationSet id="1" contentType="video" par="16:9" maxWidth="1280" maxHeight="720" maxFrameRate="30000/1001" segmentAlignment="true" mimeType="video/mp4" startWithSAP="1">
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC1=eng;CC2=spa;CC3=;CC4=" />
<SegmentTemplate initialization="$RepresentationID$Header.m4s" timescale="90000" duration="180980" media="$RepresentationID$-202500-i$Number$.m4s" startNumber="1" />
<Representation id="150000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="150000" />
<Representation id="350000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="350000" />
<Representation id="600000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="600000" />
<Representation id="900000_33" width="480" height="270" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="900000" />
<Representation id="1400000_33" width="640" height="360" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1400000" />
<Representation id="2400000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="2400000" />
<Representation id="3000000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3000000" />
<Representation id="3400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3400000" />
<Representation id="3900000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3900000" />
<Representation id="4400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4400000" />
</AdaptationSet>

position 2051 - when currentTime reaches this point we have buffered through position 2093 and suddenly start loading content from period 22 at position 2045
<Period id="19" duration="PT0H0M15.181S">
<BaseURL>https://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2020111900024306/</BaseURL>
<AdaptationSet id="1" contentType="video" par="16:9" maxWidth="1280" maxHeight="720" maxFrameRate="30000/1001" segmentAlignment="true" mimeType="video/mp4" startWithSAP="1">
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC1=eng;CC2=spa;CC3=;CC4=" />
<SegmentTemplate initialization="$RepresentationID$Header.m4s" timescale="90000" duration="170795" media="$RepresentationID$-202500-i$Number$.m4s" startNumber="1" />
<Representation id="150000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="235500" />
<Representation id="350000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="485500" />
<Representation id="600000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="798000" />
<Representation id="900000_33" width="480" height="270" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1173000" />
<Representation id="1400000_33" width="640" height="360" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1782000" />
<Representation id="2400000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3048000" />
<Representation id="3000000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3798000" />
<Representation id="3400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4298000" />
<Representation id="3900000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4923000" />
<Representation id="4400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="5548000" />
</AdaptationSet>

<Period id="20" duration="PT0H0M15.181S">
<BaseURL>https://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2020100700005706/</BaseURL>
<AdaptationSet id="1" contentType="video" par="16:9" maxWidth="1280" maxHeight="720" maxFrameRate="30000/1001" segmentAlignment="true" mimeType="video/mp4" startWithSAP="1">
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC1=eng;CC2=spa;CC3=;CC4=" />
<SegmentTemplate initialization="$RepresentationID$Header.m4s" timescale="90000" duration="170795" media="$RepresentationID$-202500-i$Number$.m4s" startNumber="1" />
<Representation id="150000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="150000" />
<Representation id="350000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="350000" />
<Representation id="600000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="600000" />
<Representation id="900000_33" width="480" height="270" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="900000" />
<Representation id="1400000_33" width="640" height="360" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1400000" />
<Representation id="2400000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="2400000" />
<Representation id="3000000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3000000" />
<Representation id="3400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3400000" />
<Representation id="3900000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3900000" />
<Representation id="4400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4400000" />
</AdaptationSet>

<Period id="21" duration="PT0H7M43S">
<AssetIdentifier schemeIdUri="urn:org:dashif:asset-id:2013" />
<BaseURL>https://edge-vod.spectrum.net/DASH_DRM/NBCU0425766000563007/</BaseURL>
<Representation id="150000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="150000" />
<Representation id="350000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="350000" />
<Representation id="600000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="600000" />
<Representation id="900000_33" width="480" height="270" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="900000" />
<Representation id="1400000_33" width="640" height="360" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1400000" />
<Representation id="2400000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="2400000" />
<Representation id="3000000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3000000" />
<Representation id="3400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3400000" />
<Representation id="3900000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3900000" />
<Representation id="4400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4400000" />

position 2545
<Period id="22" duration="PT0H0M30.129S">
<BaseURL>https://dai-scope010.timewarnercable.com/DASH_VOD/NBCU2021020200005406/</BaseURL>
<AdaptationSet id="1" contentType="video" par="16:9" maxWidth="1280" maxHeight="720" maxFrameRate="30000/1001" segmentAlignment="true" mimeType="video/mp4" startWithSAP="1">
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC1=eng;CC2=spa;CC3=;CC4=" />
<SegmentTemplate initialization="$RepresentationID$Header.m4s" timescale="90000" duration="180780" media="$RepresentationID$-202500-i$Number$.m4s" startNumber="1" />
<Representation id="150000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="150000" />
<Representation id="350000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="350000" />
<Representation id="600000_33" width="320" height="180" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="600000" />
<Representation id="900000_33" width="480" height="270" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="900000" />
<Representation id="1400000_33" width="640" height="360" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="1400000" />
<Representation id="2400000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="2400000" />
<Representation id="3000000_33" width="960" height="540" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3000000" />
<Representation id="3400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3400000" />
<Representation id="3900000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="3900000" />
<Representation id="4400000_33" width="1280" height="720" sar="1:1" frameRate="30000/1001" codecs="avc1.4d4020" scanType="progressive" bandwidth="4400000" />
</AdaptationSet>
}}

@ismena
Copy link
Contributor

ismena commented Mar 2, 2021

Hi @caridley
Thanks for reporting. Could you try v3.0.8? We had some ad+period related fixes in 3.0, you case could be affected.

@caridley
Copy link
Contributor Author

caridley commented Mar 2, 2021

@ismena This has been observed with v3.0.8

@ismena
Copy link
Contributor

ismena commented Mar 2, 2021

Understood.
Do you have the manifest hosted anywhere for an accessible repro?
Feel free to send info to [email protected] privately.

@caridley
Copy link
Contributor Author

caridley commented Mar 3, 2021

This problem seems occurs on about 5% of period transition in VOD DAI streams. Sometime the player successfully jumps the large gap introduced by loading segments from a later period. Other times it gets stuck in a buffering state.

I tried to reproduce this problem at the same position in the same manifest using a charles proxy local map to serve the manifest attached to this ticket, but the problem did not occur. Seems you just have to play a bunch of ads until the problem occurs. Will send instructions for accessing streams later today.

@ismena
Copy link
Contributor

ismena commented Mar 3, 2021

I can access your stream - thanks for setting this up!
What I'm observing - at some point the player gets stuck in a buffering state (from your report I'm assuming it's a period boundary). I don't see a clear link to ads - it gets stuck during the main presentation as well for me.

We'll investigate!

@caridley
Copy link
Contributor Author

caridley commented Mar 4, 2021

What we see is that on a period transition within an ad beak it starts loading content way out in the future. But the player doesn't fail until ~40 secnds later when it reaches the start of the humongous gap which may not occur until after we return to main content. We are using a forward buffer goal of 40 sec for VOD.

@ismena
Copy link
Contributor

ismena commented Mar 4, 2021

Yeah, I've seen the buffering gap too.
I saw a repro this morning, but - of course - when I actually sat down to look into it later in the day, the repro didn't happen :/

You know what would be super helpful - if you see it happening on your side, could you send me a snapshot of the manifest for the period that was supposed to be played (so, if we got stuck between periods 6 and 7, send the manifest text for the 7th) and the first segment of that period. I'll try to grab the same on my part, if I see the repro.

With that, I'll be able to look at the timestamps on the media vs the manifest text and see if something's up with the content.
What we often see with this buffering pattern is a mismatch between the media and the manifest with regards to the timeline.

@caridley
Copy link
Contributor Author

caridley commented Mar 4, 2021

@ismena I think the information that you are asking for is already available for the originally reported occurance in attachments to the original report and the 1st two comments.

I am doubtful about it being a problem with the content because playing through the same position in the exact same manifest does not always reproduce the problem, and we do not see this problem with shaka 2.5.x

@joeyparrish
Copy link
Member

The position in the presentation timeline is dictated by the timestampOffset attribute of SourceBuffer, which is fed in turn by timestampOffset in our SegmentReference object.

If there's a giant gap, either:

  1. the media is in the wrong place
  2. or we skipped a bunch of media

If we skipped a bunch, that points to StreamingEngine, SegmentIndex, or SegmentIterator.

If it's in the wrong place, that means either:

  1. some parameter in the manifest is wrong
  2. or we calculated the timestampOffset incorrectly from correct inputs

If we calculated timestampOffset incorrect from correct inputs, but did it only sometimes, that would imply a race condition or something similar in DashParser or PeriodCombiner.

There's a lot of assumptions above, and a tree of possibilities to explore. The first thing to verify, IMO, is whether or not the media after the giant gap is in the correct place.

@ismena
Copy link
Contributor

ismena commented Mar 4, 2021

Huh, interesting.
If it doesn't happen with v2.5, but happens intermittently in v3, I would suspect a race condition in our period flattening logic.
Let me try to hunt the repro down again, and walk through the possibilities @joeyparrish suggested.

@caridley You've provided some of what I asked in the third comment. Let me see if I got that right:
In your example, we have played through 2093, which is the end of period 19? And then started buffering at 2545, skipping period 20? Is that correct?

@ismena
Copy link
Contributor

ismena commented Mar 4, 2021

Investigation notes so far:
I got another repro, and it looks like something might be going on when we adapt between variants and (possibly) construct a SegmentIndex.
I see that after the switch, the timestamp offset on the video source buffer is equal to the duration of the first period, so we essentially skipped the rest of the period after a switch and started fetching the next period instead.
Potential culprits: streaming engine or the segment index logic.
Will look more into this tomorrow, if I can squeeze it in, or early next week if I can't.

@ismena
Copy link
Contributor

ismena commented Mar 9, 2021

I have a fix for this in code review. Thanks for your patience! :)

@joeyparrish joeyparrish added type: bug Something isn't working correctly and removed needs triage labels Mar 9, 2021
@shaka-bot shaka-bot added this to the v3.1 milestone Mar 9, 2021
@caridley
Copy link
Contributor Author

caridley commented Mar 9, 2021

@ismena would you like me to take your proposed fix for a test drive?

@ismena
Copy link
Contributor

ismena commented Mar 10, 2021

Please do. If it's checked in today (which hopefully it will be, I just have a few minor comments to address), it'll be on our nightly page by EOD. I'll let you know when it's ready.

@ismena ismena self-assigned this Mar 10, 2021
@ismena
Copy link
Contributor

ismena commented Mar 10, 2021

...and done!

@caridley Try your content on our nightly page, feel free to reopen this bug if there are issues.

If you wanna plug the fix into your source instead to test, the comment from @shaka-bot references the commit. This fix will be part of v3.1

@joeyparrish
Copy link
Member

I will also cherry-pick the fix to v3.0.9.

@caridley
Copy link
Contributor Author

Using the master branch with this fix I was able to play through 10 of our VOD ad breaks without seeing this problem. Looks like it's fixed! Thanks @ismena

@ismena
Copy link
Contributor

ismena commented Mar 11, 2021

Glad to hear.
It was a freakish rounding error that reproed once in a blue moon :/

joeyparrish pushed a commit that referenced this issue Mar 11, 2021
Fixes #3191

Change-Id: I4f2c6b3e2e67a6db1bfb71815a5ce80a3584e41e
joeyparrish pushed a commit that referenced this issue Mar 11, 2021
Fixes #3191

Backported to v2.5.x

Change-Id: I4f2c6b3e2e67a6db1bfb71815a5ce80a3584e41e
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
Development

No branches or pull requests

4 participants