Subtitles in HLS streams

[anuragkalia]

I have a question regarding how subtitles are behaving in an HLS stream. If my subtitles playlist is fragmented, it is interpreting the time of subtitles relative to the segmentStart time.

For example, let the fragment start at 4s and in individual cue in vtt file be:

00:00:49.320 --> 00:00:50.400
Sir, the media has arrived.

Then, the cue is added to video.textTrack at (4 + 49.32)s = 53.32s

This seems very counter-intuitive to me and wanted to confirm if this is a bug.

[TheModMaker]

This is currently working as intended, but that will probably change in #1563. Cue times are relative to the segment to mirror how media segments work.

[joeyparrish]

It is, essentially, a bug. Here's the history:

Segmented WebVTT in and of itself has no spec'd way of interpreting the timestamps. DASH IOP says that segmented WebVTT in DASH should be wrapped in MP4, which does have a specified way of interpreting timestamps. When we added support for segmented WebVTT in text form in DASH, we arrived at a consensus with some of our partners.

Then we added HLS support, which has the opposite way of interpreting timestamps in segmented WebVTT text.

In essence, HLS is the only spec that says how this should work, and we're doing it wrong right now. We will have to make a breaking change to the way we interpret text WebVTT in DASH, as well as some refactoring, all of which is painful, and we haven't done any of it yet.

[anuragkalia]

Thank you for the history. Really helped in my poking around. And I have a doubt there.

I referred to this document: DASH-IF Interoperability: Guidelines for Implementations.

The relevant section seems to be [6.4.5]:

Side-loaded TTML or WebVTT subtitles or caption files can be used by some players including dash.js. Such files can be indicated in the manifest like:

<AdaptationSet contentType="text" mimeType="application/ttml+xml” lang="swe">
  <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
   <Representation id="xml_swe" bandwidth="1000">
      <BaseURL>sub_swe_short.xml</BaseURL>
   </Representation>
</AdaptationSet>

Only one file for the full period is permitted, practically limiting this use case to non-live content. Such external files are assumed to have a timeline aligned with the Period, so that TTML time 00:00:00.000 corresponds to the start of the Period. The presentation time offset is expected to be not presented, and if present, expected to be ignored by the DASH client. The same applies to side-loaded WebVTT files. In that case, the @mimeType is text/vtt. If segmented subtitles are needed, such as for live sources, segments shall be packaged in ISOBMFF containers, in TTML or WebVTT format, with timing according to [29]. In particular, this means that the TTML timing inside the segments is with respect to the media timeline. [emphasis mine]".

But another section of the document says:

The media timeline origin for tracks in ISO Media files is defined to be zero. Each Representation is assigned a presentation time offset, either by the value of the attribute Representation@presentationTimeOffset or by default set to 0.

So to me, it seems like side-loaded subtitles files are supposed to be interpreted as beginning at presentation time offset. But what does the phrase "with respect to the media timeline" mean here? More specifically, I did not figure how it is to be interpreted as beginning with a segment when media timeline origin again starts at presentation time offset.

[anuragkalia]

Okay, my doubt has cleared. Going to the actual document at [29] (Timed Text and Other Visual Overlays in ISO Base Media File Format), it is mentioned specifically that WebVTT is supposed to be at segment level and non-overlapping. But TTML is at the period level. @joeyparrish Is this the discrepancy that you were referring to?

I am ready to work on this. I just do not have any sample DASH streams with segmented subtitles in WebVTT format. Where can I find it?

[TheModMaker]

That document refers to text embedded in MP4, not segmented text. There are no documents describing segmented text since the IOP section 6.4.5 says you shouldn't do it. Again, we decided to have the times be segment-relative since that was the consensus with some partners at the time.

The emphasis you added in 6.4.5 is talking about how the cues within TTML are relative to the Period instead of relative to the segment. But when embedded in MP4 this is well-defined in the spec (ISO/IEC 14496-30).

[shaka-bot]

@anuragkalia Does this answer all your questions? Can we close the issue?

[anuragkalia]

Yes, it points me to the right direction. I will have to read more about dash and the different kind of streams possible. You can close the ticket for now. Thank you!