Can't play 4k content on Xbox using WebView2

[massicottem]

Have you read the FAQ and checked for duplicate open issues?

Yes

If the problem is related to FairPlay, have you read the tutorial?

N/A

What version of Shaka Player are you using?

4.10.9

Can you reproduce the issue with our latest release version?

Yes

Can you reproduce the issue with the latest code from main?

Yes

Are you using the demo app or your own custom app?

Both

If custom app, can you reproduce the issue using our demo app?

Yes

What browser and OS are you using?

We use a WebView2 (Edge Chromium) inside a UWP on Xbox. "Xbox One" is manually added to the user-agent. If not manually added, the user-agent is the same as Edge Chromium on Windows.

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

N/A

What are the manifest and license server URIs?

We found the problem while trying to play our own content, but we were able to reproduce with Sintel.
https://storage.googleapis.com/shaka-demo-assets/sintel-mp4-only/dash.mpd

What configuration are you using? What is the output of player.getConfiguration()?

{
    "drm": {
        "retryParameters": {
            "maxAttempts": 2,
            "baseDelay": 1000,
            "backoffFactor": 2,
            "fuzzFactor": 0.5,
            "timeout": 30000,
            "stallTimeout": 5000,
            "connectionTimeout": 10000
        },
        "servers": {
            "com.widevine.alpha": "",
            "com.microsoft.playready": "",
            "com.apple.fps.1_0": ""
        },
        "clearKeys": {},
        "advanced": {},
        "delayLicenseRequestUntilPlayed": false,
        "persistentSessionOnlinePlayback": false,
        "persistentSessionsMetadata": [],
        "logLicenseExchange": false,
        "updateExpirationTime": 1,
        "preferredKeySystems": [
            "com.widevine.alpha",
            "com.microsoft.playready"
        ],
        "keySystemsMapping": {},
        "parseInbandPsshEnabled": true,
        "minHdcpVersion": "",
        "ignoreDuplicateInitData": true
    },
    "manifest": {
        "retryParameters": {
            "maxAttempts": 5,
            "baseDelay": 1000,
            "backoffFactor": 2,
            "fuzzFactor": 0.5,
            "timeout": 30000,
            "stallTimeout": 5000,
            "connectionTimeout": 10000
        },
        "availabilityWindowOverride": null,
        "disableAudio": false,
        "disableVideo": false,
        "disableText": false,
        "disableThumbnails": false,
        "defaultPresentationDelay": 0,
        "segmentRelativeVttTiming": false,
        "raiseFatalErrorOnManifestUpdateRequestFailure": false,
        "continueLoadingWhenPaused": true,
        "dash": {
            "clockSyncUri": "",
            "ignoreDrmInfo": false,
            "disableXlinkProcessing": false,
            "xlinkFailGracefully": false,
            "ignoreMinBufferTime": false,
            "autoCorrectDrift": true,
            "initialSegmentLimit": 1000,
            "ignoreSuggestedPresentationDelay": false,
            "ignoreEmptyAdaptationSet": false,
            "ignoreMaxSegmentDuration": false,
            "keySystemsByURI": {
                "urn:uuid:1077efec-c0b2-4d02-ace3-3c1e52e2fb4b": "org.w3.clearkey",
                "urn:uuid:e2719d58-a985-b3c9-781a-b030af78d30e": "org.w3.clearkey",
                "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed": "com.widevine.alpha",
                "urn:uuid:9a04f079-9840-4286-ab92-e65be0885f95": "com.microsoft.playready",
                "urn:uuid:79f0049a-4098-8642-ab92-e65be0885f95": "com.microsoft.playready"
            },
            "sequenceMode": false,
            "enableAudioGroups": false,
            "multiTypeVariantsAllowed": true,
            "useStreamOnceInPeriodFlattening": false,
            "updatePeriod": -1,
            "enableFastSwitching": true
        },
        "hls": {
            "ignoreTextStreamFailures": false,
            "ignoreImageStreamFailures": false,
            "defaultAudioCodec": "mp4a.40.2",
            "defaultVideoCodec": "avc1.42E01E",
            "ignoreManifestProgramDateTime": false,
            "ignoreManifestProgramDateTimeForTypes": [],
            "mediaPlaylistFullMimeType": "video/mp2t; codecs=\"avc1.42E01E, mp4a.40.2\"",
            "useSafariBehaviorForLive": true,
            "liveSegmentsDelay": 3,
            "sequenceMode": true,
            "ignoreManifestTimestampsInSegmentsMode": false,
            "disableCodecGuessing": false,
            "disableClosedCaptionsDetection": false,
            "allowLowLatencyByteRangeOptimization": true
        },
        "mss": {
            "sequenceMode": false,
            "keySystemsBySystemId": {
                "9a04f079-9840-4286-ab92-e65be0885f95": "com.microsoft.playready",
                "79f0049a-4098-8642-ab92-e65be0885f95": "com.microsoft.playready"
            }
        }
    },
    "streaming": {
        "retryParameters": {
            "maxAttempts": 5,
            "baseDelay": 1000,
            "backoffFactor": 2,
            "fuzzFactor": 0.5,
            "timeout": 30000,
            "stallTimeout": 5000,
            "connectionTimeout": 10000
        },
        "rebufferingGoal": 2,
        "bufferingGoal": 10,
        "bufferBehind": 30,
        "evictionGoal": 1,
        "ignoreTextStreamFailures": false,
        "alwaysStreamText": false,
        "startAtSegmentBoundary": false,
        "gapDetectionThreshold": 0.5,
        "gapJumpTimerTime": 0.25,
        "durationBackoff": 1,
        "safeSeekOffset": 5,
        "stallEnabled": true,
        "stallThreshold": 1,
        "stallSkip": 0.1,
        "useNativeHlsForFairPlay": true,
        "inaccurateManifestTolerance": 2,
        "lowLatencyMode": false,
        "autoLowLatencyMode": false,
        "forceHTTP": false,
        "forceHTTPS": false,
        "preferNativeHls": false,
        "updateIntervalSeconds": 1,
        "dispatchAllEmsgBoxes": false,
        "observeQualityChanges": false,
        "maxDisabledTime": 30,
        "parsePrftBox": false,
        "segmentPrefetchLimit": 0,
        "prefetchAudioLanguages": [],
        "disableAudioPrefetch": false,
        "disableTextPrefetch": false,
        "disableVideoPrefetch": false,
        "liveSync": {
            "enabled": false,
            "targetLatency": 0.5,
            "targetLatencyTolerance": 0.5,
            "maxPlaybackRate": 1.1,
            "minPlaybackRate": 0.95,
            "panicMode": false,
            "panicThreshold": 60,
            "dynamicTargetLatency": {
                "enabled": false,
                "stabilityThreshold": 60,
                "rebufferIncrement": 0.5,
                "maxAttempts": 10,
                "maxLatency": 4,
                "minLatency": 1
            }
        },
        "allowMediaSourceRecoveries": true,
        "minTimeBetweenRecoveries": 5,
        "vodDynamicPlaybackRate": false,
        "vodDynamicPlaybackRateLowBufferRate": 0.95,
        "vodDynamicPlaybackRateBufferRatio": 0.5,
        "infiniteLiveStreamDuration": false,
        "preloadNextUrlWindow": 30,
        "loadTimeout": 30,
        "clearDecodingCache": false,
        "dontChooseCodecs": false,
        "shouldFixTimestampOffset": false
    },
    "mediaSource": {
        "codecSwitchingStrategy": "smooth",
        "forceTransmux": false,
        "insertFakeEncryptionInInit": true
    },
    "offline": {
        "usePersistentLicense": true,
        "numberOfParallelDownloads": 5
    },
    "abr": {
        "enabled": true,
        "useNetworkInformation": true,
        "defaultBandwidthEstimate": 1000000,
        "switchInterval": 8,
        "bandwidthUpgradeTarget": 0.85,
        "bandwidthDowngradeTarget": 0.95,
        "restrictions": {
            "minWidth": 0,
            "maxWidth": null,
            "minHeight": 0,
            "maxHeight": null,
            "minPixels": 0,
            "maxPixels": null,
            "minFrameRate": 0,
            "maxFrameRate": null,
            "minBandwidth": 0,
            "maxBandwidth": null,
            "minChannelsCount": 0,
            "maxChannelsCount": null
        },
        "advanced": {
            "minTotalBytes": 128000,
            "minBytes": 16000,
            "fastHalfLife": 2,
            "slowHalfLife": 5
        },
        "restrictToElementSize": false,
        "restrictToScreenSize": false,
        "ignoreDevicePixelRatio": false,
        "clearBufferSwitch": false,
        "safeMarginSwitch": 0,
        "cacheLoadThreshold": 20,
        "minTimeToSwitch": 0
    },
    "autoShowText": 3,
    "preferredAudioLanguage": "",
    "preferredAudioLabel": "",
    "preferredTextLanguage": "",
    "preferredVariantRole": "",
    "preferredTextRole": "",
    "preferredAudioChannelCount": 6,
    "preferredVideoHdrLevel": "AUTO",
    "preferredVideoLayout": "",
    "preferredVideoLabel": "",
    "preferredVideoCodecs": [],
    "preferredAudioCodecs": [],
    "preferForcedSubs": false,
    "preferSpatialAudio": false,
    "preferredDecodingAttributes": [],
    "restrictions": {
        "minWidth": 0,
        "maxWidth": null,
        "minHeight": 0,
        "maxHeight": null,
        "minPixels": 0,
        "maxPixels": null,
        "minFrameRate": 0,
        "maxFrameRate": null,
        "minBandwidth": 0,
        "maxBandwidth": null,
        "minChannelsCount": 0,
        "maxChannelsCount": null
    },
    "playRangeStart": 0,
    "playRangeEnd": null,
    "textDisplayer": {
        "captionsUpdatePeriod": 0.25
    },
    "cmcd": {
        "enabled": false,
        "sessionId": "",
        "contentId": "",
        "rtpSafetyFactor": 5,
        "useHeaders": false,
        "includeKeys": []
    },
    "cmsd": {
        "enabled": true,
        "applyMaximumSuggestedBitrate": true,
        "estimatedThroughputWeightRatio": 0.5
    },
    "lcevc": {
        "enabled": false,
        "dynamicPerformanceScaling": true,
        "logLevel": 0,
        "drawLogo": false
    },
    "ads": {
        "customPlayheadTracker": true,
        "skipPlayDetection": true,
        "supportsMultipleMediaElements": false
    }
}

What did you do?

Start a content that is available in 4k.

What did you expect to happen?

In our app: the content should automatically play in the best available quality, 4k here.

In Shaka's demo app: we should be able to select 4k as the playback resolution.

What actually happened?

In our app: The content is playing in 1080p. A warning in the console reads "Xbox: Error detecting screen size, default screen size 1920x1080."

In Shaka's demo app: We can't select 4k as the playback resolution. A warning in the console reads "Xbox: Error detecting screen size, default screen size 1920x1080."

Before upgrading our app from using a WebView to using a WebView2, the 4k playback was working without any problem.

As we set manually add "Xbox One" in the user-agent, detectMaxHardwareResolution() knows that we are on an Xbox and try to call Windows.Media.Protection.ProtectionCapabilities() and Windows.Media.Protection.ProtectionCapabilityResult, which is working inside a regular WebView. That said, Microsoft change the way to call the native Windows APIs inside a WebView2. We need to manually add a special project called WinRTAdapter in our solution and to call the Windows APIs from the JavaScript side, we need to call chrome.webview.hostObjects.sync.Windows instead of Windows. The WinRTAdapter should be configured to expose Windows.Media.Protection.ProtectionCapabilities() and Windows.Media.Protection.ProtectionCapabilityResultMore details here : https://learn.microsoft.com/en-us/microsoft-edge/webview2/how-to/winrt-from-js.

The quick fix would be to first check if Windows is defined before try to call the Windows APIs, if it's not defined, try to define const Windows = chrome.webview.hostObjects.sync.Windows; and then, proceed normally. If it fails, it will default 1080p as it already does. By doing this, the maximum resolution detection will work well inside a WebView and a WebView2 (if the dev added "Xbox One" to the user-agent and added the WinRTAdapter project).

Are you planning send a PR to fix it?

Yes

[massicottem]

@joeyparrish Do you have an estimate of when the fix will be released? 🤔

[joeyparrish]

Whenever @avelad or I do the cherry-picks and push one... it doesn't take long, but I'm a bit preoccupied with an internal project right now.