playlists with 'usable' keystatus feature

[BucherTomas]

This is a feedback to the recently added feature of using only playlists with usable keystatus from #1460 in version 3.9.0 so apologies it does not follow the ticket template.

It is a welcome addition to the library, thank you, but it does not appear too robust at the current stage.

Here is a test stream where HD tracks require hardware-backed Widevine decryption, so all-in-all an emulation of somewhat strict real-world stream conditions:

https://videojs-http-streaming.netlify.app/?debug=true&autoplay=false&muted=false&fluid=true&minified=false&sync-workers=false&liveui=true&llhls=true&url=https%3A%2F%2F1798253129.ssl.cdn.cra.cz%2FUHD%2Ffull%2Fhddrm.ism%2Fmpdmod%2F.mpd&type=application%2Fdash%2Bxml&keysystems=%7B%0A%20%20%22com.widevine.alpha%22%3A%20%7B%0A%20%20%20%20%22url%22%3A%20%22https%3A%2F%2Fdrm-widevine-licensing.axtest.net%2FAcquireLicense%3Faxdrmmessage%3DeyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb21fa2V5X2lkIjoiNjVGM0JCNjItQUUwMS00MThGLUFCNkUtRTlBNTgwOUU3MEIxIiwibWVzc2FnZSI6eyJjb250ZW50X2tleV91c2FnZV9wb2xpY2llcyI6W3sibmFtZSI6Imhkc2VjIiwicGxheXJlYWR5Ijp7Im1pbl9kZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOjMwMDAsImFuYWxvZ192aWRlb19vdXRwdXRfcHJvdGVjdGlvbnMiOlt7ImNvbmZpZ19kYXRhIjoiQUFBQUF3PT0iLCJpZCI6IkMzRkQxMUM2LUY4QjctNEQyMC1CMDA4LTFEQjE3RDYxRjJEQSJ9XSwiYW5hbG9nX3ZpZGVvX29wbCI6MTUwLCJjb21wcmVzc2VkX2RpZ2l0YWxfdmlkZW9fb3BsIjo1MDAsInVuY29tcHJlc3NlZF9kaWdpdGFsX3ZpZGVvX29wbCI6MzAwfSwid2lkZXZpbmUiOnsiaGRjcCI6IjEuMCIsImRldmljZV9zZWN1cml0eV9sZXZlbCI6IkhXX1NFQ1VSRV9ERUNPREUifX0seyJuYW1lIjoic2RzZWMiLCJwbGF5cmVhZHkiOnsibWluX2RldmljZV9zZWN1cml0eV9sZXZlbCI6MjAwMCwiYW5hbG9nX3ZpZGVvX291dHB1dF9wcm90ZWN0aW9ucyI6W3siY29uZmlnX2RhdGEiOiJBQUFBQXc9PSIsImlkIjoiQzNGRDExQzYtRjhCNy00RDIwLUIwMDgtMURCMTdENjFGMkRBIn1dLCJhbmFsb2dfdmlkZW9fb3BsIjoxNTAsImNvbXByZXNzZWRfZGlnaXRhbF92aWRlb19vcGwiOjUwMCwidW5jb21wcmVzc2VkX2RpZ2l0YWxfdmlkZW9fb3BsIjoyNTB9LCJ3aWRldmluZSI6eyJkZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOiJTV19TRUNVUkVfREVDT0RFIn19LHsicGxheXJlYWR5Ijp7Im1pbl9kZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOjIwMDB9LCJuYW1lIjoiYXVkaW8iLCJ3aWRldmluZSI6eyJkZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOiJTV19TRUNVUkVfQ1JZUFRPIn19XSwidmVyc2lvbiI6MiwidHlwZSI6ImVudGl0bGVtZW50X21lc3NhZ2UiLCJjb250ZW50X2tleXNfc291cmNlIjp7ImlubGluZSI6W3sidXNhZ2VfcG9saWN5Ijoic2RzZWMiLCJpZCI6ImY1YmYzM2FjLTA4NDQtMGM4MS1kNjIzLTAxOWM4N2ZlMTM2MCIsIml2IjoiVDNIZUw4emIyTExnb2dkVmI5a1h3UT09In0seyJ1c2FnZV9wb2xpY3kiOiJoZHNlYyIsImlkIjoiZjRlYThkYjgtNzZhZS05YzhjLTJhMjEtOTg4MmQ3NjNiYzdkIiwiaXYiOiJpRXVLTEFRMElvYVpzZ3Voc0s1SENRPT0ifSx7InVzYWdlX3BvbGljeSI6ImF1ZGlvIiwiaWQiOiJmZjE0ZmM5Yy1iOGM3LTYwNjctY2Q0Yy1hY2YyOTQzNjU4NWEiLCJpdiI6Im1rUTA2bENFdG45YkVLYStzSUpMUGc9PSJ9XX0sImxpY2Vuc2UiOnsiZHVyYXRpb24iOjYwNDgwMCwid2lkZXZpbmUiOnsiaW5jbHVkZV9hbGxfZW50aXRsZWRfa2V5cyI6dHJ1ZX19fSwidmVyc2lvbiI6MX0.rxbmZ8KWPdqigtrvXhvB76d2NKjSMFuHGBt5U8rsEnM%22%0A%20%20%7D%0A%7D&buffer-water=false&exact-manifest-timings=false&pixel-diff-selector=false&network-info=false&dts-offset=false&override-native=true&preload=auto&mirror-source=false&forced-subtitles=false

The same stream configuration works as expected on desktop Chrome with Widevine L3 in players such as Shaka or Bitmovin and representations with higher resolution than 576p are removed from the stream. By the way, L2 or L1 are also becoming available for desktop Chrome on Windows with "com.widevine.alpha.experiment" keysystem, so this should be also something for you to take into account in the near future, but I digress.

The same stream fails in VHS once HD tracks are selected, because their keyID f4ea8db876ae9c8c2a219882d763bc7d is apparently still used and not filtered out even though according to EME logger extension, the MediaKeyStatusMap includes just video SD keyID f5bf33ac08440c81d623019c87fe1360 and audio track keyID ff14fc9cb8c76067cd4cacf29436585a, both tagged as usable. License server does not provide in this case any response for f4ea8db876ae9c8c2a219882d763bc7d at all, which is recommended, so it is not present even tagged as output-restricted for example. You can see for yourselves with the stream above.

The test stream configuration above is for Widevine. To expand on this implementation also for PlayReady, it behaves differently and if a keyID is unusable in current context (for example requesting SL3000 license for HD tracks from a SL2000 device), the license server may instead return status code 500 for the license associated with HD tracks but all the others will be fine. In current state, this is a fatal failure for the player. Instead the player should continue, attempt to utilize all the remaining usable licenses and again remove unusable representations from the stream for which the license request failed. PlayReady might also already need the com.microsoft.playready.recommendation and com.microsoft.playready.recommendation.3000 keysystems.

Also a question, does the implemented logic react to ad-hoc changes in keystatus? For example:

1. HD tracks require HDCP 2.0
2. The player starts streaming on integrated notebook display which supports HDCP 2.2, so all keyIDs and associated tracks are usable
3. While streaming, an external monitor capable of HDCP 1.0 only is connected, CDM detects that and flags the keyID for HD tracks as output-restricted
4. Player should react to it and switch to remaining available representations for SD tracks only
   The same should be applicable also in reverse when you disconnect the external monitor which should enable the HD tracks again.

[adrums86]

Thank you very much for this feedback @BucherTomas we should support the configuration from your test stream, as the new feature will only enable playlists with keyIds that have a matching keyId from the MediaKeyStatusMap. I found one of the problems and created a draft PR to fix it, seemed to be a matching problem with the keyIds in the provided manifest being uppercase and the keyIds from the MediaKeyStatusMap always being lowercase. You can try the build here.

Obviously there will be room for future improvement, so your continued feedback is very much appreciated. The current implementation should react to ad-hoc changes to the keystatus as long as the MediaKeyStatusMap reflects the status change on a keystatuseschange event.

[BucherTomas]

Thank you for the quick reaction @adrums86 . Our team provided PR videojs/mpd-parser#157 to mpd-parser some time ago for this exact reason, so it has not occured to me that it could be the same situation.

It works in this preview version and it could be therefore merged to the main branch. I am going to test some more multikey stream examples with various license restrictions next week with a computer configuration with multiple monitors.

But I would also put some emphasis on handling the PlayReady environment as well, because otherwise this filtering solution of usable tracks is still not ready for real usage with multiple DRM systems in its current state.

You can use the same stream example, just add the keysystem for PlayReady and simply replace the license server domain to drm-playready-licensing.axtest.net, the rest of the url remains the same. This #1424 needs to be fixed first.

Thank you in advance.

[adrums86]

@BucherTomas Merged #1466 which should fix the widevine example. Will work on some verification of the PlayReady environment as soon as I'm able. Regarding the changeType issue, I found it's actually reproducible on unencrypted assets in Edge as well, so they can be looked at in parallel. Hopefully we can have all these addressed relatively quickly. As always, feedback and collaboration is appreciated! 🍻

[BucherTomas]

I tested it some more as I promised previously. Generally, the feature works with Widevine as expected, happy to see that the player tries to react to ad-hoc keystatus changes. But:

• it would help to get the debug log in sync with actual keystatus
• there appear to be some cases where the resulting behavior is not perfect

This time I used a license where HD tracks are usable on desktop Chrome with Widevine L3 but just require HDCP 2.0. SD tracks are without restrictions. Please use https://videojs-http-streaming.netlify.app/?debug=true&autoplay=false&muted=false&fluid=true&minified=false&sync-workers=false&liveui=true&llhls=true&url=https%3A%2F%2F1798253129.ssl.cdn.cra.cz%2FUHD%2Ffull%2Fhddrm.ism%2Fmpdmod%2F.mpd&type=application%2Fdash%2Bxml&keysystems=%7B%0A%20%20%22com.widevine.alpha%22%3A%20%7B%0A%20%20%20%20%22url%22%3A%20%22https%3A%2F%2Fdrm-widevine-licensing.axtest.net%2FAcquireLicense%3Faxdrmmessage%3DeyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb21fa2V5X2lkIjoiNjVGM0JCNjItQUUwMS00MThGLUFCNkUtRTlBNTgwOUU3MEIxIiwibWVzc2FnZSI6eyJjb250ZW50X2tleV91c2FnZV9wb2xpY2llcyI6W3sibmFtZSI6Imhkc2VjIiwicGxheXJlYWR5Ijp7Im1pbl9kZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOjIwMDAsImRpZ2l0YWxfdmlkZW9fb3V0cHV0X3Byb3RlY3Rpb25zIjpbeyJjb25maWdfZGF0YSI6IkFBQUFBUT09IiwiaWQiOiJBQkIyQzZGMS1FNjYzLTQ2MjUtQTk0NS05NzJEMTdCMjMxRTcifV0sImFuYWxvZ192aWRlb19vdXRwdXRfcHJvdGVjdGlvbnMiOlt7ImNvbmZpZ19kYXRhIjoiQUFBQUFRPT0iLCJpZCI6Ijc2MEFFNzU1LTY4MkEtNDFFMC1CMUIzLURDREY4MzZBNzMwNiJ9XSwiYW5hbG9nX3ZpZGVvX29wbCI6MjAxLCJjb21wcmVzc2VkX2RpZ2l0YWxfdmlkZW9fb3BsIjo1MDAsInVuY29tcHJlc3NlZF9kaWdpdGFsX3ZpZGVvX29wbCI6MzAwfSwid2lkZXZpbmUiOnsiZGlzYWJsZV9hbmFsb2dfb3V0cHV0Ijp0cnVlLCJoZGNwIjoiMi4wIiwiY2dtcy1hIjoibmV2ZXIiLCJkZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOiJTV19TRUNVUkVfREVDT0RFIn19LHsibmFtZSI6InNkc2VjIiwicGxheXJlYWR5Ijp7Im1pbl9kZXZpY2Vfc2VjdXJpdHlfbGV2ZWwiOjIwMDB9LCJ3aWRldmluZSI6eyJjZ21zLWEiOiJuZXZlciIsImRldmljZV9zZWN1cml0eV9sZXZlbCI6IlNXX1NFQ1VSRV9DUllQVE8ifX0seyJwbGF5cmVhZHkiOnsibWluX2RldmljZV9zZWN1cml0eV9sZXZlbCI6MjAwMH0sIm5hbWUiOiJhdWRpbyIsIndpZGV2aW5lIjp7ImRldmljZV9zZWN1cml0eV9sZXZlbCI6IlNXX1NFQ1VSRV9DUllQVE8ifX1dLCJ2ZXJzaW9uIjoyLCJ0eXBlIjoiZW50aXRsZW1lbnRfbWVzc2FnZSIsImNvbnRlbnRfa2V5c19zb3VyY2UiOnsiaW5saW5lIjpbeyJ1c2FnZV9wb2xpY3kiOiJzZHNlYyIsImlkIjoiZjViZjMzYWMtMDg0NC0wYzgxLWQ2MjMtMDE5Yzg3ZmUxMzYwIiwiaXYiOiJUM0hlTDh6YjJMTGdvZ2RWYjlrWHdRPT0ifSx7InVzYWdlX3BvbGljeSI6Imhkc2VjIiwiaWQiOiJmNGVhOGRiOC03NmFlLTljOGMtMmEyMS05ODgyZDc2M2JjN2QiLCJpdiI6ImlFdUtMQVEwSW9hWnNndWhzSzVIQ1E9PSJ9LHsidXNhZ2VfcG9saWN5IjoiYXVkaW8iLCJpZCI6ImZmMTRmYzljLWI4YzctNjA2Ny1jZDRjLWFjZjI5NDM2NTg1YSIsIml2IjoibWtRMDZsQ0V0bjliRUthK3NJSkxQZz09In1dfSwibGljZW5zZSI6eyJkdXJhdGlvbiI6NjA0ODAwLCJ3aWRldmluZSI6eyJpbmNsdWRlX2FsbF9lbnRpdGxlZF9rZXlzIjp0cnVlfX19LCJ2ZXJzaW9uIjoxfQ.d014Y5rfR_-YWrsmtG5dFYLy0MIMhZxPk0tRBkhCykM%22%0A%20%20%7D%0A%7D&buffer-water=false&exact-manifest-timings=false&pixel-diff-selector=false&network-info=false&dts-offset=false&override-native=true&preload=auto&mirror-source=false&forced-subtitles=false

Start the playback on a notebook with integrated HDCP 2.0+ display. In this case all playlists are correctly enabled, i.e. all key ids are usable but the log states

enabling playlist 0-placeholder-uri-0 because key ID f5bf33ac08440c81d623019c87fe1360 is usable
enabling playlist 1-placeholder-uri-1 because key ID f5bf33ac08440c81d623019c87fe1360 is usable
enabling playlist 2-placeholder-uri-2 because key ID f5bf33ac08440c81d623019c87fe1360 is usable
enabling playlist 3-placeholder-uri-3 because key ID f5bf33ac08440c81d623019c87fe1360 is usable

playlists 4 and 5 with key ID f4ea8db876ae9c8c2a219882d763bc7d for HD tracks are not listed in the log at all but they are available in the Representations selector in the demo player as they should. All playlists should be therefore accounted for also in the log.

• Once I connect external display with HDCP 1.0 during playback, the player attempts to get rid of HD tracks and starts spewing to the debug log for all playlists and all video key ids

excluding playlist *-placeholder-uri-* because the key ID *** doesn't exist in the keyStatusMap or is not usable
just to log a while later that playlists 0-3 are usable. This cycle then repeats indefinitely as long as that monitor is connected. I assume this is because iterations of keyStatusMap, but it shouldn't ideally report during that phase that all playlists are excluded as this would normally cause the playback to stop.

• When the external monitor is connected while an HD representation is being rendered, the Representations selector switches to displaying just SD playlists which is correct, but the playback typically pauses here and waitingforkey event is fired, because player does not actually force-switch to the nearest available SD playlist automatically. This is crucial.

• When the external monitor is connected while playlist 3 with width 1024 is manually selected, i.e. one of the SD tracks, the Representations selector still displays also playlists for HD tracks and when a switch to one of the HD playlists or other SD playlists is attempted, the player actually does not adapt and still renders playlist 3. The map between usable playlists and available representations for the user to choose from seems to be broken in this case.

• When the external monitor is connected, the Representations selector may become completely empty even though the stream remains playable. I unfortunately don't have further details, it occured just once.

• When the playback is started while the external monitor is connected, the Representations selector correctly displays just SD playlists. Once the monitor is disconnected and playback is resumed on integrated notebook monitor, the HD playlists are correctly enabled and the player even adapts to an HD playlist, but the Representations selector then displays just them instead of adding them next to the SD playlists and displaying all of them together. The debug log correctly states that all playlists are enabled.

Also it appears that there is no handling of status-pending keystatus that might occur when for example the HDCP handshake takes a bit longer and the CDM is not yet capable of deciding whether given key is usable or not.

Thank you in advance for making the feature more robust.

[BucherTomas]

There were some more tests planned by the team, particularly with PlayReady, but it appears to have lost its momentum. Have you managed to take a look at some of it?

There is an additional issue with PlayReady, likely related to this topic where even streams with one non-restrictive single key for all dash representations are being postprocessed and HD and higher representations are filtered out, even though both the license and codec are supported within the given environment. The console outputs:

excluding playlist *** because the key ID *** doesn't exist in the keyStatusMap or is not usable for all playlists
followed by
enabling non-HD playlist *** because all playlists were excluded due to non-usable key IDs for multiple playlist ids up to 576p.

However, seeing the enumeration from EME Logger extension, MediaKeyStatusMap returned that key id is "usable" so this result is of course wrong. This concerns just PlayReady, Widevine works fine in this situation.

We can see it with our own streams, but here is one public PlayReady stream from Microsoft where HD and higher representations are accessible in other tested players, just not in Video.js.

https://v3-12-2--videojs-http-streaming.netlify.app/?debug=true&autoplay=false&muted=false&fluid=false&minified=false&sync-workers=false&liveui=true&llhls=true&url=https%3A%2F%2Fprofficialsite.origin.mediaservices.windows.net%2Fc51358ea-9a5e-4322-8951-897d640fdfd7%2Ftearsofsteel_4k.ism%2Fmanifest(format%3Dmpd-time-csf)&type=application%2Fdash%2Bxml&keysystems=%7B%0A%20%20%22com.microsoft.playready%22%3A%20%7B%0A%20%20%20%20%22url%22%3A%20%22https%3A%2F%2Ftest.playready.microsoft.com%2Fservice%2Frightsmanager.asmx%3Fcfg%3D(persist%3Afalse%2Csl%3A150)%22%0A%20%20%7D%0A%7D

Thank you.

[adrums86]

@BucherTomas thank you for your persistence here. The core contributors for the entire video.js org have been quite busy lately so much of this work has been getting pushed back. That said, we do have some planned stabilization/improvement work in the near-ish future that this work can absolutely be a part of. I unfortunately cannot provide a solid timeline however. I'm going to add a point to revisit and improve this usable logic as part of that effort. I'll update this issue once it's in progress, your help is greatly appreciated!
If you happen to have the time and feel like taking a look, most of the code is here in VHS and below for filtering based on keyId. This works in conjunction with eventing from videojs-contrib-eme here. There has been some improvement to the eventing from contrib-eme lately so I think this filtering logic can be improved quite a bit overall in conjunction with the contrib-eme work.

[BucherTomas]

@adrums86 I believe the issue might stem from key ID endianness. PlayReady is based on Little Endian, so the order of key ID must be handled differently.

There is a hint that I overlooked in the console with that Microsoft stream example:

KeyStatus 'usable' with key ID e11a656fe4db3444bcb4690d1564c41c added to the keyStatusMap
excluding playlist 0-placeholder-uri-0 because the key ID 6f651ae1dbe44434bcb4690d1564c41c doesn't exist in the keyStatusMap or is not usable

Both lines are referencing the same KID, but each time it is using different endianness. That's why it likely doesn't match and the excludeNonUsablePlaylistsByKeyId function kicks in for PlayReady but not for Widevine.

[adrums86]

Good catch, I thought I had handled the endian inconsistencies between keysystems but apparently not. When I'm able to dig into this, I'll start there.