Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for negative initial timestamps #3498

Closed
dimitry-unified-streaming opened this issue Jun 30, 2021 · 9 comments
Closed

Support for negative initial timestamps #3498

dimitry-unified-streaming opened this issue Jun 30, 2021 · 9 comments
Labels
priority: P3 Useful but not urgent status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone

Comments

@dimitry-unified-streaming
Copy link

dimitry-unified-streaming commented Jun 30, 2021

Have you read the FAQ and checked for duplicate open issues?
Yes.

What version of Shaka Player are you using?
v3.1.1 (debug)

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Unknown, I assume v3.1.1 (debug) is master?

Are you using the demo app or your own custom app?
Demo app, at https://shaka-player-demo.appspot.com/demo

If custom app, can you reproduce the issue using our demo app?
n/a

What browser and OS are you using?
Chrome 91.0.4472.114/macOS 11.4: hangs
Chrome 91.0.4472.124/Windows 10.0.19043.1052: hangs
Edge 91.0.864.59/Windows 10.0.19043.1052: hangs
Firefox 89.0.2 /macOS 11.4: works
Firefox 89.0.2/Windows 10.0.19043.1052: works
Safari 14.1.1/macOS 11.4: hangs

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
n/a

What are the manifest and license server URIs?
https://live.unified-streaming.com/shaka-issue-8845/nodrm.ism/.mpd?vbegin=346&vend=1668
(no DRM applied, this should play right away)

Direct demo link:
https://shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=en-US;uilang=en-US;asset=https://live.unified-streaming.com/shaka-issue-8845/nodrm.ism/.mpd?vbegin=346&vend=1668;panel=CUSTOM%20CONTENT;build=debug_compiled

What configuration are you using? What is the output of player.getConfiguration()?
No special configuration, using defaults from https://shaka-player-demo.appspot.com/

What did you do?
Attempt to play the media.

What did you expect to happen?
Media should have started playing. However, it only does so on Firefox, not on Chrome, Edge or Safari. I haven't tried embedded players, but I'm assuming Chromium-derived and Safari-derived browsers will fail identically.

What actually happened?
On Chrome and Safari browsers, the media does not start playing, and a spinner is shown.

The last messages in the console are:

Calling switch_(), bandwidth=4669 kbps
player.js:5095 switch_
gap_jumping_controller.js:226 Ignoring large gap at 0 size 1.907336

However, as far as I can determine, there is no such gap. The manifest has audio and video which are slightly misaligned due to (probably) hiccups in the encoder, and this causes the audio to start 0.013666s later than the video.

For reference, the full manifest:

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Unified Streaming Platform  (version=1.11.1-24062) -->
<MPD
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="urn:mpeg:dash:schema:mpd:2011"
  xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
  type="static"
  mediaPresentationDuration="PT22M2.866334S"
  maxSegmentDuration="PT2S"
  minBufferTime="PT10S"
  profiles="urn:mpeg:dash:profile:isoff-live:2011">
  <Period
    id="1"
    duration="PT22M2.866334S">
    <BaseURL>dash/</BaseURL>
    <AdaptationSet
      id="1"
      group="1"
      contentType="audio"
      lang="fr"
      segmentAlignment="true"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="mp4a.40.2"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011"
        value="2" />
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" />
      <SegmentTemplate
        timescale="48000"
        presentationTimeOffset="16589456"
        duration="92160"
        startNumber="181"
        initialization="nodrm-$RepresentationID$.dash"
        media="nodrm-$RepresentationID$-$Number$.m4s">
      </SegmentTemplate>
      <Representation
        id="1=128000"
        bandwidth="128000">
      </Representation>
    </AdaptationSet>
    <AdaptationSet
      id="2"
      group="2"
      contentType="video"
      par="71:40"
      minBandwidth="331000"
      maxBandwidth="1402000"
      maxWidth="1024"
      maxHeight="576"
      segmentAlignment="true"
      sar="1:1"
      mimeType="video/mp4"
      startWithSAP="1">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" />
      <SegmentTemplate
        timescale="250000"
        presentationTimeOffset="86403416"
        eptDelta="-3416"
        duration="480000"
        startNumber="181"
        initialization="nodrm-$RepresentationID$.dash"
        media="nodrm-$RepresentationID$-$Number$.m4s">
      </SegmentTemplate>
      <Representation
        id="video=331000"
        bandwidth="331000"
        width="426"
        height="240"
        codecs="avc1.640015"
        scanType="progressive">
      </Representation>
      <Representation
        id="video=634000"
        bandwidth="634000"
        width="640"
        height="360"
        codecs="avc1.64001E"
        scanType="progressive">
      </Representation>
      <Representation
        id="video=1402000"
        bandwidth="1402000"
        width="1024"
        height="576"
        codecs="avc1.64001F"
        scanType="progressive">
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

The starting segments for both audio and video are number 181 (as indicated by startNumber=181):

The baseMediaDecodeTime of audio segment 181 is 16589456/48000 = 345.613666s, while the baseMediaDecodeTime of video segment 181 is 86400000/250000 = 345.600000s. So there is a slight misalignment of 0.013666s, but definitely not a 'gap' of 1.907336s!

Note that due to DASH-IF implementation guidelines for the restricted timing model, in particular 9.2.1. Necessary segment references in static presentations, we have adjusted the presentationTimeOffset of the video track to ensure both video and audio is available at the start of the presentation. E.g.:

  • Audio segment 181 starts at 345.613666s, and in the AdaptationSet it has timescale="48000" presentationTimeOffset="16589456", so this starts at exactly 345.613666s too, while
  • Video segment 181 starts at 345.600000s, and in the AdaptationSet it has timescale="250000" presentationTimeOffset="86403416", so this starts at 345.613664s, which is as near to the audio as possible.
@dimitry-unified-streaming
Copy link
Author

Note that I saw some similarities with #3189 ("DASH VOD manifest with large offset at the start can't be played", still open) and #3451 ("Don't jump over the gap at the beginning before the video is played so that the video poster doesn't disappear", closed), but these are (I think) subtly different issues.

@TheModMaker
Copy link
Contributor

What I think is happening is the first video segment is being skipped. Because the timestamp is 86400000, but the presentationTimeOffset is 86403416, so the first timestamp is negative. The browser will evict frames up to the next key frame, so this may evict up to the next segment, which results in the gap.

If you adjust the presentationTimeOffset backward slightly so both audio and video start at or after 0, the gap will be gone. Or you can set streaming.jumpLargeGaps to true.

@DimitryAndric
Copy link

What I think is happening is the first video segment is being skipped. Because the timestamp is 86400000, but the presentationTimeOffset is 86403416, so the first timestamp is negative. The browser will evict frames up to the next key frame, so this may evict up to the next segment, which results in the gap.

If you adjust the presentationTimeOffset backward slightly so both audio and video start at or after 0, the gap will be gone. Or you can set streaming.jumpLargeGaps to true.

The timing model now requires that presentationTimeOffset be set to the beginning of the part of the presentation where both audio and video are available, which is why we set it like this. Otherwise, the beginning of the presentation would have only video, and no audio. To compensate, we also add an eptDelta attribute, as specified in MPEG DASH Amendment 5:

A Representation should cover all presentation times Tp that is are greater than and equal to the value of the @presentationTimeOffset, T0. If smaller values are present, i.e. the value of the @eptDelta is negative, then presentation of the Media Segment is expected to only take place for presentation times greater than or equal to T0.

It seems that neither Shaka nor DASH.js have explicit support for eptDelta, but DASH.js has no trouble playing this manifest from the beginning (e.g. it works on all browsers). I'm unsure what it is doing differently.

@dimitry-unified-streaming
Copy link
Author

Or you can set streaming.jumpLargeGaps to true.

FWIW that works, obviously (see this link), and it displays Jumping forward 1.907336 seconds because of gap before start time of 1.907336 in the log. (But only on Chrome, Edgium and Safari; on Firefox the gap is never detected, nor is it jumped over.)

But still, reading the MPEG DASH Amendment 5 text (which ended up in in ISO/IEC 23009-1:2019 under "7.2.1 Media Presentation timeline") it seems that the idea is that if 'negative' values are present, initial frames should be decoded and skipped, and should only be shown from the presentationTimeOffset onward.

Note that the 'negative' values are really an artifact of how the player handles the timeline, as the actual segments never contain negative timestamps. I.e., the player could also subtract the period@start time instead of the presentationTimeOffset, and then the intermediate timestamps would not become negative. I believe that is how DASH.js handles this situation, as it has no trouble playing these streams (and it doesn't jump any gaps, as far as I can see).

@TheModMaker TheModMaker changed the title Hang on gap (which is not really a gap) at the start of video/audio Support for negative initial timestamps Aug 23, 2021
@TheModMaker TheModMaker added type: enhancement New feature or request priority: P3 Useful but not urgent and removed needs triage labels Aug 23, 2021
@TheModMaker TheModMaker added this to the Backlog milestone Aug 23, 2021
@avelad
Copy link
Member

avelad commented Jan 31, 2023

This is already supported using the sequence mode. Can you test with main branch? Thanks!

@avelad avelad added the status: waiting on response Waiting on a response from the reporter(s) of the issue label Jan 31, 2023
@DimitryAndric
Copy link

Using https://shaka-player-demo.appspot.com/demo/, which I assume runs the latest main, I now see a playing stream. In the console I see:

Jumping forward 0.014666 seconds because of gap before start time of 0.014666

so that is as expected, I think? In other words, it now seems to work correctly. Any idea in which version of Shaka this was fixed?

@github-actions github-actions bot removed the status: waiting on response Waiting on a response from the reporter(s) of the issue label Jan 31, 2023
@avelad
Copy link
Member

avelad commented Jan 31, 2023

@joeyparrish can you help here?

@joeyparrish
Copy link
Member

Sequence mode for HLS was introduced in v4.0.0. The demo app should be displaying the latest release, which as of yesterday, is v4.3.4.

Does this help?

@avelad avelad added the status: waiting on response Waiting on a response from the reporter(s) of the issue label Feb 1, 2023
@dimitry-unified-streaming
Copy link
Author

Thanks, that is useful information. It's mostly about when clients ask when this issue was solved, we can tell them to upgrade Shaka past v4.0.0, if possible.

@github-actions github-actions bot removed the status: waiting on response Waiting on a response from the reporter(s) of the issue label Feb 1, 2023
@avelad avelad modified the milestones: Backlog, v4.4 Feb 3, 2023
@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Apr 2, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
priority: P3 Useful but not urgent status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants