Allow the user agent to choose which media element source to play remotely

[avayvod]

Since the media element can have more than one source, the UA should be able to remote what it thinks is the best for the selected remote device.

Also maybe allow the website to specify the preferred source for remote playback (e.g. on mobile the element may have the default local playback source in SD quality while a link to an HD media could be provided for remote playback specifically).

The website could already just change the element source when it's being played remotely though.

[hober]

I think I'd prefer a design where the local UA hands the set of possible sources to the remote device, so it can make its own decision re: which to play.

[avayvod]

Yes, we should allow the remote playback device to take part. Some existing devices might not be able to accept multiple sources so I'd avoid excluding the local UA out of the process.

[tidoust]

Discussed at the F2F:
http://www.w3.org/2016/05/24-webscreens-minutes.html#item07

PROPOSED RESOLUTION: Anton to update spec to be clear that entire source list is considered for availability (if possible) and remote playback

ACTION: Investigate HTML source selection algorithm to decide if it is applicable, possibly on @foolip

[foolip]

I would caution against changing the resource selection algorithm, that works by picking the first source that works, and doesn't ever switch once playback is in progress. In particular it doesn't ever look ahead or do any scoring to pick a best resource. (Note that this is different from how picture works, which also has source children.)

You will also have sites that don't use source elements but instead use canPlayType to set the source, and they would need some API to achieve the same thing.

I guess the problem is that we might be playing using MSE locally, but the remote isn't a browser at all and instead uses DASH or a plain URL? I suppose what is needed is more information about the remote in RemotePlaybackAvailability so that an alternative resource can be provided in connect()?

Whatever the API looks like, there will be fun questions like what to do if the remote resource has a different duration, or if it doesn't have the same set of tracks as the local one.

[avayvod]

This was specified in #49. The UA either MUST use currentSrc for availability and remote playback or MUST use all the available media resources and pick one of them for remote playback.

[foolip]

Any thoughts about the issues in #7 (comment)? I don't quite see why this is allowed to be implementation-specific, can't the spec simply say what will be implemented?

[avayvod]

@foolip do you mean that in the latter case, the UA should also choose the first media resource that works?

I guess we could also say that currentSrc might change to the selected resource then.
Duration and tracks (if supported) should be messaged back by the remote playback device if it's media flinging/remoting or determined by the UA in case of mirroring.

Any other issues I missed? canPlayType() refers to the local playback. Using it for remote playback seems problematic as the user agent should already select the resource by the time the user selects the device. Perhaps, we could change that and canPlayType in "connected" state should return the info for the connected device.

[foolip]

I mean that the algorithm for picking a resource can be spelled out in detail unless diverging behavior is desirable or perhaps unavoidable. If it's left undefined, then sharing tests between implementations won't be possible because any behavior is allowed.

The trouble with canPlayType is that sites can rely entirely on that instead of appending source child elements, in fact I'd guess that this is pretty common. In that case the resource selected for local playback will be the only one known to the "what should I play remotely" algorithm but it might be a resource that doesn't work remotely.

How to make this work well depends on what the (internal) API for the remote side looks like. Will it be possible to try playing a resource and fall back to another if it fails to decode? That's what happens with source child elements, in particular when there's no type attribute it's the only way to find the resource that you support.

My concern is that the source child elements will have two processing models:

1. The current model, where each source element is processed in tree order (with a "pointer" to keep track of which one to try next) and the first one that could work is tried. If it's a totally unsupported resource, the next source child is tried.
2. A model for remote, where at a single point in time, all source child elements are collected and turned into some representation sent to the remote. The best of these is used, and if it doesn't work the second best is tried, and so on.

(Nobody has spelled out that remote would work like that, but it's tempting to do that.)