[Question] does RxPlayer support fragmented Web VTT? #1638
Comments
JohnPaulHarold
changed the title
|
Hi, We do support fragmented webvtt. When loading the corresponding text track segments, I see that the This doesn't seem to be a valid WEBTT file as far as we understand its specification especially its file structure, as it's supposed to be starting with a Looking at a subtitle segment actually containing subtitles, the subtitle's content I see when translating into ASCII is: Maybe it's another version of the format we're not familiar with? Maybe the packaging software you're using to make segmented subtitles has an issue? |
|
Looking further into it (on other players), it seems to be in an mp4-specialized format that I didn't know of. I'm looking if it's easy to do on our side. |
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
|
OK I've made an attempt at parsing it. I deployed the current work as a demo page here: https://developers.canal-plus.com/rx-player/vttc/ so you can check on your side if it works for your case. The work is still WIP as there are some architectural questions left but that demo should be close to the final result. |
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
peaBerberian
added a commit
that referenced
this issue
Jan 28, 2025
As #1638 reported, we do not support some (or all?) ways of communicating webvtt subtitles in mp4 files. I initially assumed that webvtt embedded in an mp4 file worked like for TTML (a format we're more used to): the plain subtitle format directly inserted in an `mdat` box. Turns out, some contents (like https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd), actually rely on the metadata from other ISOBMFF boxes (mainly the tfdt and trun boxes) to provide timing information with the `mdat` reserved for text, identifiers and style information in a new binary format following the ISOBMFF way of doing things (new webvtt-specific boxes). Weirdly enough, that format does not look at all like the WEBVTT we're used to beside the fact that it uses the same identifier and "settings" concept. --- As now our subtitle parser has to have the context of the whole mp4 file (and not of the mp4 segment), and as that parser also has to rely on state (a `timescale` value) provided by an initialization segment, I had to update the architecture of how subtitles are communicated: they can now be communicated as string or `BufferSource` types (the latter leading the text encoding detection), and a supplementary `timescale` argument (defaulting to `1`) is now always provided to parsers. The vast majority of parsers now do not make use of that `timescale` value which is kind of ugly though, we may want to find a better solution.
|
@peaBerberian thank you for this, and my apologies for the delay in replying. I've tested the same stream on your demo page and the wvtt options now appear to be rendering correctly. I'll see if I can checkout the branch on the accompanying PR also and test with some other streams. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello.
note: I've put this as a question, in case I'm doing something wrong, but it might also be an issue. I can reword as that distinction is established.
I am trying this stream, https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel-wvtt.ism/.mpd
It has an array of text track choices, but crucially, it offers text tracks in fragmented mode, but also "sidecar". When I test this stream in the RxPlayer demo https://developers.canal-plus.com/rx-player/ the player instance shows all 10 text track options (2 x 5 languages). This can also be seen in the browser repl, looking at
rxPlayer.getAvailableTextTracks(). The first 5 text tracks are the fragmented Web VTT options, while the rest are sidecar equivalents.RxPlayer seems to play the sidecar variants fine, but does not render the first five.
My question is, does RxPlayer support this fragmented style of captions?
some digging around...
via an
errorevent handler, this is thrown,which seems to correspond to this area of the code
rx-player/src/parsers/texttracks/webvtt/html/parse_webvtt_to_div.ts
Lines 44 to 46 in 169e6d6
If I put a breakpoint around that area in the bundled code, the
textarg is"\u0000\u0000\u0000\bvtte". At a guess, the parser throws as the string does not begin withWEBVTT?The text was updated successfully, but these errors were encountered: