Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add liveSync configuration to catch up on live streams #5304

Merged
merged 7 commits into from
Jun 22, 2023
Merged

feat: Add liveSync configuration to catch up on live streams #5304

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d64060a
feat: Add liveSync configuration
avelad Jun 14, 2023
000e98f
Change default value to 1.1
avelad Jun 14, 2023
a8bc358
Fix comparison
avelad Jun 14, 2023
ef70a61
Fix demo
avelad Jun 15, 2023
b1e42c3
Change comment
avelad Jun 17, 2023
a56bd4d
Merge branch 'main' into live-sync
avelad Jun 21, 2023
5c4e099
Merge branch 'main' into live-sync
avelad Jun 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions demo/common/message_ids.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,9 @@ shakaDemo.MessageIds = {
LCEVC_LOG_LEVEL: 'DEMO_LCEVC_LOG_LEVEL',
LCEVC_SECTION_HEADER: 'DEMO_LCEVC_SECTION_HEADER',
LIVE_SEGMENTS_DELAY: 'DEMO_LIVE_SEGMENTS_DELAY',
LIVE_SYNC: 'DEMO_LIVE_SYNC',
LIVE_SYNC_MAX_LATENCY: 'DEMO_LIVE_SYNC_MAX_LATENCY',
LIVE_SYNC_PLAYBACK_RATE: 'DEMO_LIVE_SYNC_PLAYBACK_RATE',
LOG_LEVEL: 'DEMO_LOG_LEVEL',
LOG_LEVEL_DEBUG: 'DEMO_LOG_LEVEL_DEBUG',
LOG_LEVEL_INFO: 'DEMO_LOG_LEVEL_INFO',
Expand Down
12 changes: 11 additions & 1 deletion demo/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,17 @@ shakaDemo.Config = class {
.addNumberInput_(MessageIds.MAX_DISABLED_TIME,
'streaming.maxDisabledTime')
.addNumberInput_(MessageIds.SEGMENT_PREFETCH_LIMIT,
'streaming.segmentPrefetchLimit');
'streaming.segmentPrefetchLimit')
.addBoolInput_(MessageIds.LIVE_SYNC,
'streaming.liveSync')
.addNumberInput_(MessageIds.LIVE_SYNC_MAX_LATENCY,
'streaming.liveSyncMaxLatency',
/* canBeDecimal= */ true,
/* canBeZero= */ false)
.addNumberInput_(MessageIds.LIVE_SYNC_PLAYBACK_RATE,
'streaming.liveSyncPlaybackRate',
/* canBeDecimal= */ true,
/* canBeZero= */ false);

if (!shakaDemoMain.getNativeControlsEnabled()) {
this.addBoolInput_(MessageIds.ALWAYS_STREAM_TEXT,
Expand Down
3 changes: 3 additions & 0 deletions demo/locales/en.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,9 @@
"DEMO_LIVE": "Live",
"DEMO_LIVE_SEARCH": "Live",
"DEMO_LIVE_SEGMENTS_DELAY": "Live segments delay",
"DEMO_LIVE_SYNC": "Live sync",
"DEMO_LIVE_SYNC_MAX_LATENCY":"Max latency for live sync",
"DEMO_LIVE_SYNC_PLAYBACK_RATE":"Playback rate for live sync",
"DEMO_LOG_LEVEL": "Log Level",
"DEMO_LOG_LEVEL_DEBUG": "Debug",
"DEMO_LOG_LEVEL_INFO": "Info",
Expand Down
12 changes: 12 additions & 0 deletions demo/locales/source.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -543,6 +543,18 @@
"description": "The name of a configuration value.",
"message": "Live segments delay"
},
"DEMO_LIVE_SYNC": {
"description": "The name of a configuration value.",
"message": "Live sync"
},
"DEMO_LIVE_SYNC_MAX_LATENCY": {
"description": "The name of a configuration value.",
"message": "Max latency for live sync"
},
"DEMO_LIVE_SYNC_PLAYBACK_RATE": {
"description": "The name of a configuration value.",
"message": "Playback rate for live sync"
},
"DEMO_LOG_LEVEL": {
"description": "The name of a configuration value.",
"message": "Log Level"
Expand Down
17 changes: 16 additions & 1 deletion externs/shaka/player.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1078,7 +1078,10 @@ shaka.extern.ManifestConfiguration;
* observeQualityChanges: boolean,
* maxDisabledTime: number,
* parsePrftBox: boolean,
* segmentPrefetchLimit: number
* segmentPrefetchLimit: number,
* liveSync: boolean,
* liveSyncMaxLatency: number,
* liveSyncPlaybackRate: number
* }}
*
* @description
Expand Down Expand Up @@ -1193,6 +1196,18 @@ shaka.extern.ManifestConfiguration;
* ahead of playhead in parallel.
* If <code>0</code>, the segments will be fetched sequentially.
* Defaults to <code>0</code>.
* @property {boolean} liveSync
* Enable the live stream sync against the live edge by changing the playback
* rate. Defaults to <code>false</code>.
* Note: on some SmartTVs, if this is activated, it may not work or the sound
* may be lost when activated.
* @property {number} liveSyncMaxLatency
* Maximum acceptable latency, in seconds. Effective only if liveSync is
* true. Defaults to <code>1</code>.
* @property {number} liveSyncPlaybackRate
* Playback rate used for latency chasing. It is recommended to use a value
* between 1 and 2. Effective only if liveSync is true. Defaults to
* <code>1.2</code>.
avelad marked this conversation as resolved.
Show resolved Hide resolved
* @exportDoc
*/
shaka.extern.StreamingConfiguration;
Expand Down
53 changes: 53 additions & 0 deletions lib/player.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2275,6 +2275,11 @@ shaka.Player = class extends shaka.util.FakeEventTarget {
'arbitrary language initially');
}

if (this.isLive() && this.config_.streaming.liveSync) {
const onTimeUpdate = () => this.onTimeUpdate_();
this.loadEventManager_.listen(mediaElement, 'timeupdate', onTimeUpdate);
}

this.fullyLoaded_ = true;

// Wait for the 'loadedmetadata' event to measure load() latency.
Expand Down Expand Up @@ -2627,6 +2632,11 @@ shaka.Player = class extends shaka.util.FakeEventTarget {
fullyLoaded.reject(abortedError);
return Promise.resolve(); // Abort complete.
}).chain(() => {
if (this.isLive() && this.config_.streaming.liveSync) {
const onTimeUpdate = () => this.onTimeUpdate_();
this.loadEventManager_.listen(mediaElement, 'timeupdate', onTimeUpdate);
}

this.fullyLoaded_ = true;
});
}
Expand Down Expand Up @@ -5691,6 +5701,49 @@ shaka.Player = class extends shaka.util.FakeEventTarget {
}
}

/**
* Callback for liveSync
*
* @private
*/
onTimeUpdate_() {
// If the live stream has reached its end, do not sync.
if (!this.isLive()) {
return;
}
const seekRange = this.seekRange();
if (!Number.isFinite(seekRange.end)) {
return;
}
const currentTime = this.video_.currentTime;
if (currentTime > seekRange.start) {
// Bad stream?
avelad marked this conversation as resolved.
Show resolved Hide resolved
return;
}
const liveSyncMaxLatency = this.config_.streaming.liveSyncMaxLatency;
const liveSyncPlaybackRate = this.config_.streaming.liveSyncPlaybackRate;
const playbackRate = this.video_.playbackRate;
const latency = seekRange.end - this.video_.currentTime;
let offset = 0;
// HLS in SRC= doesn't update every time the seekrange, so we need calculate
// a safe offset
if (this.loadMode_ == shaka.Player.LoadMode.SRC_EQUALS) {
const buffered = this.video_.buffered;
if (buffered.length > 0) {
const bufferedEnd = buffered.end(buffered.length - 1);
offset = Math.max(liveSyncPlaybackRate, bufferedEnd - seekRange.end);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are two different units fed into Math.max: a playback rate and a duration in seconds. What is the intent here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the timeupdate event is not fired every frame, instead it takes a bit of time, sometimes even one second. The idea here is to avoid reaching the live edge in less than a second. If the playbackrate is 3 for example, and we have 3 seconds left for the live point, it would take us a second to reach it, so we want to avoid this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option is put the min in 0, but I'd prefer the current option

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your explanation makes sense, but I don't find the code clear enough on its own without this conversation. And I will definitely forget.

Here's a comment that I think might help, to replace the one above:

// In src= mode, the seek range isn't updated frequently enough, so we need to fudge
// the latency number with an offset.  The playback rate is used as an offset, since that
// is the amount we catch up 1 second of accelerated playback.

I still don't understand the other duration, though. Why do you subtract bufferedEnd - seekRange.end? That should be negative if we've fallen behind, right?

-------|---------------|-------------|--------------|------
seekRange.start   currentTime   bufferedEnd   seekRange.end

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bufferedEnd can greater than seekRange.end when the current time is near to live edge.

I’ll add your comment tomorrow. Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Math.max involving the bufferedEnd - seekRange.end parameter still doesn't make sense to me.

Here's a concrete scenario. Let's say your latency goal is 3s. You want to be within 3s of the end of the seek range.

currentTime = 10
seekRange.end = 15
bufferedEnd = 18
liveSyncPlaybackRate = 1.1

latency = seekRange.end - currentTime = 5

offset = Math.max(liveSyncPlaybackRate, bufferedEnd - seekRange.end)
offset = Math.max(1.1, 18 - 15)
offset = 3

latency - offset = 2, which is less than our target of 3. So even though we're 5s behind the end of the seek range, we don't accelerate playback. And because we've buffered past the end of the seek range, we could accelerate playback very safely.

So it seems to me that in the cases where bufferedEnd > seekRange.end, the offset value has the opposite effect of what it should do.

Am I crazy?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, but if we take into account Theodore's comment, the event should trigger at most every 250ms, so it would still be fine. (https://developer.mozilla.org/en-US/docs/Web/API/HTMMLediaElement/timeupdate_event)

Note: this is for valid, because the src= for live should only be used in Safari.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4Hz is the minimum frequency, not the maximum. The maximum is 66Hz, which would be one call every 15ms.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of the maximum, we have no problem :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to MDN, the lowest frequency you can expect for the timeupdate event is around 4 hz.
So taking an entire second's worth of fast-playback as an offset might be overkill, if the expectation is that this is being called at least 4 times a second. Even if you want to be safe and assume that system load is high, liveSyncPlaybackRate / 2 might still work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my tests this does not always work on SmartTV, I prefer to be more conservative in this case.

}
}

if ((latency - offset) > liveSyncMaxLatency) {
if (playbackRate != liveSyncPlaybackRate) {
this.trickPlay(liveSyncPlaybackRate);
}
} else if (playbackRate !== 1 && playbackRate !== 0) {
this.cancelTrickPlay();
}
}

/**
* Callback from Playhead.
*
Expand Down
3 changes: 3 additions & 0 deletions lib/util/player_configuration.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,9 @@ shaka.util.PlayerConfiguration = class {
// When low latency streaming is enabled, segmentPrefetchLimit will
// default to 2 if not specified.
segmentPrefetchLimit: 0,
liveSync: false,
liveSyncMaxLatency: 1,
liveSyncPlaybackRate: 1.2,
};

// WebOS, Tizen, and Chromecast have long hardware pipelines that respond
Expand Down