How can I query for the active codec and bandwidth being played in a mixed-codec, multi-period manifest?

[willdharris]

Have you read the Tutorials?
Yes

Have you read the FAQ and checked for duplicate open issues?
Yes

If the question is related to FairPlay, have you read the tutorial?

N/A

What version of Shaka Player are you using?
4.9.0 and main

What browser and OS are you using?
Chrome Version 125.0.6422.113 (Official Build) (arm64)
macOS Ventura 13.6

Please ask your question

My question is related to Shaka's period flattening and representation matching. During playback, when a variant has multiple codecs that it could change between, is there a way to query for the true codec and bandwidth being played?

• I have a DAI manifest (DASH, static, multi-period) with representations of different video codecs per period.
• When Shaka flattens the periods, it matches representations from each period and creates a single outputStream.
• The single outputStream could include multiple different codecs and bandwidths that could be played.
• That outputStream eventually becomes a variant track with a single video codec and bandwidth value.
• The alternate, matched representations that the variant could be playing are obfuscated.

During playback, is there a way to access the obfuscated representations during the periods in which they are being played?

Example

• My manifest has 3 periods: pre-roll-1-ad-1, pre-roll-1-ad-2, 0.
• Periods pre-roll-1-ad-1 and pre-roll-1-ad-2 only have hev and avc representations, they do not have dvhe representations.
• Period 0 has dvhe representations.
• I want to play the Dolby Vision version of the content, so I configure the player with preferredVideoCodes = ['dvhe'].
• Since the pre-roll periods do not contain dvhe representations, I see hev representations from the pre-roll periods have been matched to the dvhe outputStreams as the alternate representations.
  
• During playback of the pre-rolls, if I query for the active variant track player.getVariantTracks().find((track) => track.active), I will see that a dvhe codec variant is playing. I know that is not accurate because the manifest does not have dvhe representations in the pre-roll periods. If I inspect the network for the segments being download and buffered, I can confirm that it's the hev segments being played.

In the case of the above screenshot, depending on the presentation time, this variant labeled with a dvhe codec could be playing an hev codec with a bandwidth of 4.2mb or it could be playing the actual dvhe codec with a bandwidth of 5.8mb. Is there a way to access the true codec and bandwidths being played?

[avelad]

@theodab Since you did this part, I'll let you answer the question.

[shaka-bot]

@willdharris Does this answer all your questions? If so, would you please close the issue?

[shaka-bot]

@willdharris Does this answer all your questions? If so, would you please close the issue?

[theodab]

There is no built-in way to check this. You could use shaka.Player.getManifest to determine this from the manifest, but relying on that player method is not suggested as the manifest structure is not part of the public interface and might change between versions.
So I'll just write a player method for it, I guess.

[caridley]

I think the mediaqualitychanged events have the information that you are looking for

    interface MediaQualityInfo {
      audioSamplingRate?: number;
      bandwidth: number;
      codecs: string;
      contentType: string;
      frameRate?: number;
      height?: number;
      mimeType?: number;
      channelsCount?: number;
      pixelAspectRatio?: string;
      width?: number;
    }

[caridley]

but the solution in #6825 looks cleaner

[willdharris]

Thanks @caridley. I hadn't looked at the mediaqualitychanged event for this yet, that looks helpful.

@theodab The proposed getCurrentPlaybackInfo method would be great to have. Thanks for working on that!

[avelad]

@caridley It seems that mediaqualitychanged has many shortcomings... It only works for MP4 with initialization segments and only for DASH.

[avelad]

We've added getFetchedPlaybackInfo and improved mediaqualitychanged to support HLS and MSS, is that enough for you?

[willdharris]

Thanks @avelad! These options resolve my question here. If it's an easy add, it would be nice to also have the video width and height returned on the video object in getFetchedPlaybackInfo. Otherwise, we can get that from the mediaqualitychanged event.

[theodab]

Easy enough to add.

...Though everyone who could give me code review is on vacation until next week at the earliest, so it might be some time before it can be merged.