fix(cast): Use cast platform APIs in MediaCapabilties polyfill #4727

JulianDomingo · 2022-11-17T21:35:22Z

See #4726 for more context.

This allows Cast devices to properly filter stream variants with a resolution surpassing that of the device's capabilities.

We place the fix in the MediaCapabilities polyfill since it's intended to be the right way to check for anything related to platform support.

HDR support checks will require eotf=smpte2048, as indicated in #2813 (comment). Specifically, a {hev|hvc}1.2 profile is only an indication of an HDR transfer function, but may be a non-HDR 10-bit color stream. Take for instance this publicly available hls.js HEVC stream (https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8) which contains stream variants with the hvc1.2.4.L123.B0 codecs, but with a resolution < 4K:

#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=1996813,BANDWIDTH=2194827,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=960x540,FRAME-RATE=60.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v14/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=6135662,BANDWIDTH=6694727,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=1920x1080,FRAME-RATE=60.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v18/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=4831572,BANDWIDTH=5458398,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=1920x1080,FRAME-RATE=60.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v17/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=3471811,BANDWIDTH=4110269,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=1920x1080,FRAME-RATE=60.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v16/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=2666432,BANDWIDTH=2795200,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=1280x720,FRAME-RATE=60.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v15/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=1169111,BANDWIDTH=1256754,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=768x432,FRAME-RATE=30.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v13/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=859838,BANDWIDTH=932269,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=640x360,FRAME-RATE=30.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v12/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=552728,BANDWIDTH=579426,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=480x270,FRAME-RATE=30.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v11/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=345440,BANDWIDTH=371212,CODECS="hvc1.2.4.L123.B0,ec-3",RESOLUTION=416x234,FRAME-RATE=30.000,CLOSED-CAPTIONS="cc",AUDIO="a3",SUBTITLES="sub1"
v10/prog_index.m3u8

In Cast, the platform can distinguish between the two by explicitly providing the transfer function; it uses smpte2048 ("PQ") because this is the "basis of HDR video formats..." (https://en.wikipedia.org/wiki/Perceptual_quantizer). Below you can see the platform correctly distinguishing between an HDR-capable and non-HDR capable HDMI sink device (using a 4K-capable Chromecast):

canDisplayType() results on a 1920x1080 ASUS monitor:

[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=1920; height=1080; eotf="smpte2084"') => false
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=3840; height=2160; eotf="smpte2084"') => false
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=1920; height=1080') => true
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=3840; height=2160') => false
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"') => true
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0; eotf="smpte2084"') => false
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=1920; height=1080; eotf="smpte2084"') => false
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=3840; height=2160; eotf="smpte2084"') => false
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=1920; height=1080') => true
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=3840; height=2160') => false
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"') => true
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0; eotf="smpte2084"') => false

canDisplayType() results on a 4K HDR TCL 2017 Roku TV (43"):

New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=1920; height=1080; eotf="smpte2084"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=3840; height=2160; eotf="smpte2084"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=1920; height=1080') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"; width=3840; height=2160') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hev1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hev1.2.4.L153.B0; eotf="smpte2084"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=1920; height=1080; eotf="smpte2084"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=3840; height=2160; eotf="smpte2084"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=1920; height=1080') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"; width=3840; height=2160') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0"') => true
VM10:1 New style VP9 codec string must start with vp09. codecId:hvc1.2.4.L153.B0
[JULIAN]: canDisplayType(video/mp4; codecs="hvc1.2.4.L153.B0; eotf="smpte2084"') => true

github-actions · 2022-11-17T22:15:58Z

Incremental code coverage: 100.00%

…u TV.

…rs-in-mediacapabilitiesdecodinginfo-polyfill-for-cast-platform

lib/polyfill/media_capabilities.js

…d capitalization error for the 'framerate' parameter.

avelad · 2022-11-21T17:35:19Z

@JulianDomingo There seems to be a lot of bugs in the tests when running on Chromecast, can you review it? Thanks!

JulianDomingo · 2022-11-21T20:12:07Z

Sure thing, it looks like canDisplayType() is returning false for basic video stream configs, surprisingly. So there's no stream variants to test on - looking into it.

… Chromecast.

…ions.

joeyparrish

LGTM if spyOn doesn't need cleanup

test/polyfill/media_capabilities_unit.js

…ble. Only throw an error if the entire cast namespace is unavailable.

…omecast tests.

….__platform.__canDisplayType

…ab failures.

JulianDomingo · 2022-11-24T00:52:52Z

@avelad @joeyparrish Based on my findings with the Chromecast lab failures, the tests were failing due to cast.__platform__ && cast.__platform__.canDisplayType being unavailable (but cast is available). My solution is to fall back to the original MediaSource.isTypeSupported(contentType) if the full cast API set isn't available.

In this commit which removes the cast.__platform__ && cast.__platform__.canDisplayType check (720fe11), you can see Chromecast lab tests fail.
The subsequent commit re-instates the above check (8465a99) and the Chromecast lab tests succeed.

This obviously contradicts with my manual tests on my Chromecast devices. Maybe this has something to do with the canDisplayType binding not being "applied" yet when the tests are ran? I'm not as knowledgeable on how the unit tests / integration tests are "hooked" to the platform when lab tests are ran.

joeyparrish · 2022-11-29T00:00:10Z

lib/polyfill/media_capabilities.js

+    frameRate = undefined,
+    transferFunction = undefined,
+  }) {
+    if (!(window.cast && cast)) {


nit: I don't believe you need both of these checks. I would just say if (!window.cast).

Done, removed.

joeyparrish · 2022-11-29T00:02:07Z

lib/polyfill/media_capabilities.js

+   *     width: Describes the stream horizontal resolution in pixels.
+   *     height: Describes the stream vertical resolution in pixels.
+   *     frameRate: Describes the frame rate of the stream.
+   *     transferFunction: Describes the video transfer function supported by
+   *         the rendering capabilities of the user agent.


Since all of these come from videoConfig, I feel like you should just pass that instead of creating a new record type.

Done, thanks

joeyparrish · 2022-11-29T00:02:29Z

lib/polyfill/media_capabilities.js

+          shaka.util.Error.Category.CAST,
+          shaka.util.Error.Code.CAST_API_UNAVAILABLE);
+    } else if (!(cast.__platform__ && cast.__platform__.canDisplayType)) {
+      shaka.log.warning('Expected cast APIs to be available! Falling back to ' +


This is very strange. I'd like to help you look into that, but I don't want to hold up the PR for it.

Yeah, was held up on this for a while - the lab tests were failing without this check.

I spent a solid day on this, and it's time for me to give up for now. I managed to forward the Cast platform APIs through an iframe boundary, but not in a way that is compatible with Karma's websocket usage. :-(

test/polyfill/media_capabilities_unit.js

joeyparrish · 2022-11-29T00:09:52Z

test/polyfill/media_capabilities_unit.js

+            'codecs="hev1.2.4.L153.B0"; ' +
+            'width=512; ' +
+            'height=288; ' +
+            'framerate=23.976023976023978; ' +


This looks brittle. I don't know if we can rely on this exact amount of detail in converting a number to a string on all devices. Can you make the framerate 24 for this test instead?

Done, forgot to consider precision here.

theodab · 2022-11-29T01:28:00Z

lib/polyfill/media_capabilities.js

+   *     width: Describes the stream horizontal resolution in pixels.
+   *     height: Describes the stream vertical resolution in pixels.
+   *     frameRate: Describes the frame rate of the stream.
+   *     transferFunction: Describes the video transfer function supported by
+   *         the rendering capabilities of the user agent.


See #4726 for more context. This allows Cast devices to properly filter stream variants with a resolution surpassing that of the device's capabilities. We place the fix in the `MediaCapabilities` polyfill since it's intended to be the right way to check for anything related to platform support. HDR support checks will require `eotf=smpte2048`, as indicated in #2813 (comment). Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR transfer function, but *may* be a non-HDR 10-bit color stream. In Cast, the platform can distinguish between the two by explicitly providing the transfer function; it uses `smpte2048` (`"PQ"`) because this is the "basis of HDR video formats..." (https://en.wikipedia.org/wiki/Perceptual_quantizer).

Initial fix with temp debug logs.

aa973af

JulianDomingo added component: HLS The issue involves Apple's HLS manifest format priority: P1 Big impact or workaround impractical; resolve before feature release platform: Cast Issues affecting Cast devices labels Nov 17, 2022

JulianDomingo self-assigned this Nov 17, 2022

JulianDomingo changed the title ~~Initial fix with temp debug logs.~~ fix(polyfill): Call cast.__platform__.canDisplayType() to determine supported for MediaCapabilties.decodingInfo() polyfill in Cast devices. Nov 17, 2022

avelad linked an issue Nov 18, 2022 that may be closed by this pull request

Account for extended MIME type parameters in MediaCapabilities.decodingInfo() polyfill for Cast platform #4726

Closed

avelad added this to the v4.4 milestone Nov 18, 2022

avelad added type: bug Something isn't working correctly and removed component: HLS The issue involves Apple's HLS manifest format labels Nov 18, 2022

JulianDomingo added 4 commits November 18, 2022 11:48

Removed console debug logs after verifying eotf param on a 4K HDR Rok…

0ee3554

…u TV.

Updated comment on smpte2048 check.

4873f27

Merge branch 'main' into 4726-account-for-extended-mime-type-paramete…

2d56c99

…rs-in-mediacapabilitiesdecodinginfo-polyfill-for-cast-platform

Added existence check for transferFunction parameter.

ef09c30

joeyparrish requested changes Nov 19, 2022

View reviewed changes

lib/polyfill/media_capabilities.js Outdated Show resolved Hide resolved

lib/polyfill/media_capabilities.js Outdated Show resolved Hide resolved

JulianDomingo added 4 commits November 18, 2022 18:55

Handled review comments (iter #0) and added unit tests. Also correcte…

51a96ea

…d capitalization error for the 'framerate' parameter.

Fixed unit test linting errors.

9588678

Restored cast namespace after polyfill test executions.

b60d59a

Use window.cast instead of cast for namespace checking.

bf81981

JulianDomingo added 2 commits November 21, 2022 17:11

Sanity check to confirm issue with CDT() support on Selenium lab test…

279b029

… Chromecast.

Updated canDisplayType() checks to guard against instable implementat…

1dd0722

…ions.

joeyparrish previously approved these changes Nov 22, 2022

View reviewed changes

test/polyfill/media_capabilities_unit.js Show resolved Hide resolved

JulianDomingo dismissed joeyparrish’s stale review via 715061f November 22, 2022 02:35

JulianDomingo force-pushed the 4726-account-for-extended-mime-type-parameters-in-mediacapabilitiesdecodinginfo-polyfill-for-cast-platform branch 4 times, most recently from f625b35 to 2b8ecf9 Compare November 23, 2022 07:27

Fall back to MediaSource.isTypeSupported() if cast APIs aren't availa…

6a8cf51

…ble. Only throw an error if the entire cast namespace is unavailable.

JulianDomingo force-pushed the 4726-account-for-extended-mime-type-parameters-in-mediacapabilitiesdecodinginfo-polyfill-for-cast-platform branch from 2b8ecf9 to 6a8cf51 Compare November 23, 2022 09:16

JulianDomingo added 3 commits November 23, 2022 09:37

Removed window.cast to trigger CAST_API_UNAVAILABLE error for lab chr…

d0701ea

…omecast tests.

Verifying lab tests fail without check for cast.__platform__ and cast…

720fe11

….__platform.__canDisplayType

Re-instated deleted code from previous commit, which prevented cast l…

8465a99

…ab failures.

JulianDomingo requested review from joeyparrish and avelad November 24, 2022 00:53

avelad previously approved these changes Nov 24, 2022

View reviewed changes

joeyparrish reviewed Nov 29, 2022

View reviewed changes

theodab requested changes Nov 29, 2022

View reviewed changes

Handling PR review comments.

be1065a

JulianDomingo dismissed avelad’s stale review via be1065a November 29, 2022 03:38

theodab approved these changes Nov 29, 2022

View reviewed changes

joeyparrish approved these changes Dec 1, 2022

View reviewed changes

joeyparrish changed the title ~~fix(polyfill): Call cast.__platform__.canDisplayType() to determine supported for MediaCapabilties.decodingInfo() polyfill in Cast devices.~~ fix(cast): Use cast platform APIs in MediaCapabilties polyfill Dec 1, 2022

joeyparrish merged commit 5d6f56a into shaka-project:main Dec 1, 2022

github-actions bot mentioned this pull request Dec 1, 2022

chore(main): release 4.4.0 #4692

Merged

github-actions bot mentioned this pull request Dec 8, 2022

chore(v4.3.x): release 4.3.1 #4802

Merged

github-actions bot mentioned this pull request Dec 8, 2022

chore(v4.2.x): release 4.2.5 #4672

Merged

github-actions bot mentioned this pull request Dec 8, 2022

chore(v3.3.x): release 3.3.14 #4804

Merged

github-actions bot mentioned this pull request Dec 8, 2022

chore(v3.2.x): release 3.2.16 #4805

Merged

github-actions bot added the status: archived Archived and locked; will not be updated label Jul 25, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cast): Use cast platform APIs in MediaCapabilties polyfill #4727

fix(cast): Use cast platform APIs in MediaCapabilties polyfill #4727

JulianDomingo commented Nov 17, 2022 •

edited

Loading

github-actions bot commented Nov 17, 2022 •

edited

Loading

avelad commented Nov 21, 2022

JulianDomingo commented Nov 21, 2022

joeyparrish left a comment

JulianDomingo commented Nov 24, 2022

joeyparrish Nov 29, 2022

JulianDomingo Nov 29, 2022

joeyparrish Nov 29, 2022

theodab Nov 29, 2022

JulianDomingo Nov 29, 2022

joeyparrish Nov 29, 2022

JulianDomingo Nov 29, 2022

joeyparrish Dec 1, 2022

joeyparrish Nov 29, 2022

JulianDomingo Nov 29, 2022

theodab Nov 29, 2022

fix(cast): Use cast platform APIs in MediaCapabilties polyfill #4727

fix(cast): Use cast platform APIs in MediaCapabilties polyfill #4727

Conversation

JulianDomingo commented Nov 17, 2022 • edited Loading

github-actions bot commented Nov 17, 2022 • edited Loading

avelad commented Nov 21, 2022

JulianDomingo commented Nov 21, 2022

joeyparrish left a comment

Choose a reason for hiding this comment

JulianDomingo commented Nov 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JulianDomingo commented Nov 17, 2022 •

edited

Loading

github-actions bot commented Nov 17, 2022 •

edited

Loading