Adds a backpressure signal to the streaming protocol.

[markafoltz]

Addresses Issue #241: Add a back pressure signal for media remoting.

This adds a backpressure signal to the streaming-session-receiver-stats-event message, so the receiver can report the status of its frame buffer (if it is using one). The states are borrowed from my interpretation of w3c/remote-playback#126.

An item to discuss is how and whether to expose this to script along the lines of w3c/remote-playback#126.

---

Preview | Diff

[markafoltz]

@takumif @mounirlamouri @eric-carlson @chrisn

Notifying folks who were involved in earlier discussions around this for any feedback on the PR.

The idea is to allow the receiver of a media stream (i.e., from media remoting) to signal the sender of the media stream whether it should adjust its data rate up or down to prevent buffer overruns or underruns. Here's the timeline of previous discussions:

• Initial proposal for the Remote Playback API by @takumif: Add a remoting buffer state attribute to RemotePlayback remote-playback#126
• Discussion at TPAC 2019: https://www.w3.org/2019/09/16-webscreens-minutes.html#x02
• Resulting OSP issue to add a backpressure signal: Add a back pressure signal for media remoting #241
• Discussion at TPAC 2020: https://www.w3.org/2020/10/20-webscreens-minutes.html#a02
• Followup issue to expose this signal via the Remote Playback API (in the future): Allow adapting the bitrate to network/receiver constraints when using MSE remote-playback#132

[louaybassbouss]

Thanks @mfoltzgoogle for the PR happy to review it but I need to understand one thing: the main use case as I can see from the related issues is MSE. Are the frames decoded on the sender before they are transmitted to the receiver or the sender just transmits the media segments to the receiver without re-encoding?

[louaybassbouss]

@mfoltzgoogle the backpressure solution you provided LGTM. It added @chrisn and @mounirlamouri as reviewers and I hope we can merge it soon. @takumif @eric-carlson feel free to review as well (I couldn't add you as reviewer)

[markafoltz]

Thanks @mfoltzgoogle for the PR happy to review it but I need to understand one thing: the main use case as I can see from the related issues is MSE. Are the frames decoded on the sender before they are transmitted to the receiver or the sender just transmits the media segments to the receiver without re-encoding?

That's correct, the main use case is when the sender has pre-encoded media that is being fetched from a server or file, and wants to play it back on a remote device (agent receiving the stream in the protocol). For Web apps using the Remote Playback API, the media would be provided from the Web app to the protocol through MSE.

The sender can use the backpressure signal from the receiver to make adjustments to the send rate and how much it's buffering on the sender side. Some senders may be able to transcode the media to change the effective data rate.

It might be helpful to generate simulations showing when the signal might be sent and how it can be used to ensure uninterrupted playback on the receiver device when remoting from MSE.

[chrisn]

Apart from one minor typo, this looks good to me. I agreed with @mfoltzgoogle's comment in the meeting that feedback from prototyping could be helpful.

[louaybassbouss]

Thanks @mfoltzgoogle for the explanation and @chrisn for review. Lets wait 1-2 days for more feedback and merge this PR.

[anssiko]

LGTM.

We did not explicitly note #241 as a FPWD blocker in #249 (comment) but seems like we want to squeeze this in the FPWD so I’d suggest snapshotting that release when this PR is merged.

@tidoust, can you check we can make 18 March FPWD with this integrated i.e. builds w/o errors and passes all the publication checks? Thanks!

[markafoltz]

It sounds like we want to ensure this is in the FPWD, so I went ahead and merged the current snapshot of the PR. I will leave the issue open in case there is additional feedback from reviewers.