adding cleaned up SEI frame-type 6 logic for 608/708 captions #334

jlacivita · 2015-07-10T23:16:02Z

So here's a summary of what i've done:

detect frame type of 6 in TSDemuxer
when frame type is 6, confirm it's actually 608 captions
if so then extract the bytes typically sent in OSMF onCaptionInfo calls (it's not the entire frame data)
create an onCaptionInfo message (the defacto OSMF standard for 608 captions) with a payload that matches de-facto object { "type": "708", "data": bytes}
spit out a bunch of console messages to compare what I'm expecting to what OSMFCCLib displays

currently i'm getting almost recognizable english out of both 4 & 5, but characters (and commands) are dropped for some reason.

I'm fairly convinced that there's some problem with the NAL Unit parsing, upstream, because I'm seeing characters dropped/corrupted in both my debug log messages, as well as in the events dispatched to OSMFCCLib.

OSMFCCLib is just a tool to make sure we're parsing the packets properly. Once it works, we can discuss writing our own view or not, and cleaning up the delivery method to be something other than onCaptionInfo (maybe that logic simply belongs in the OSMF wrapper to flashls).

maxaf · 2015-07-15T00:41:26Z

@jlacivita Which stream are you using for testing? I have one with CEA-608 captions baked in, but it doesn't appear to pass through your forest of if() statements, as if it's not even the right format.

I'm testing with the following: http://cdn3.videos.bloomberg.com/btv/us/master.m3u8

mangui · 2015-07-15T15:21:45Z

@jlacivita regarding your potential NALu type 6 parsing issue.

basically the parsing processing is a follow:
TS packet => PES packet => NAL units

if you suspect a NAL parsing issue, the idea would be to binary dump the complete PES packet holding SEI NAL unit, so that we could check whether the missing bytes are present in the PES packet or not ...

so basically you add the following code here:
https://github.com/mangui/flashls/blob/dev/src/org/mangui/hls/demux/TSDemuxer.as#L473

                } else if(frame.type == 6) {
                    //dump PES packet, dump SEI NALu
                    CONFIG::LOGGING {
                        if (HLSSettings.logInfo) {
                            var ba2 : ByteArray = new ByteArray();
                            pes.data.position = pes.payload;
                            pes.data.readBytes(ba2);
                            Log.info("PES holding SEI data :" + Hex.fromArray(ba2));
                            ba2 = new ByteArray();
                            pes.data.position = frame.start;
                            pes.data.readBytes(ba2, 0, frame.length);
                            Log.info("SEI data :" + Hex.fromArray(ba2));
                        }
                    }
                }

if the missing bytes are in the PES packet, then it is a NAL parsing issue.
if not ... it could be a TS parsing issue (but I doubt ...) or something else.
Cheers,
Mangui

jlacivita · 2015-07-15T18:00:34Z

@mangui I'll take a look
@maxaf, that's a live stream... i'm not ready to switch my testing from VOD to live :) I'm using the stream from: http://www.cpcweb.com/webcasts/hls/cpcdemo.m3u8 but you hve to set up a local http proxy to server a crossdomain.xml file or it won't work.

jlacivita · 2015-07-15T21:39:21Z

Okay, i made some important discoveries.

I've been assuming ccType needed to have a value of 0 or 1, which is the value HDS streams in OSMF always used when dispatching onCaptionInfo. After searching through the NAL units, i've found that the subtitles in HLS seem to have ccType set to 2. If i change my "if" clause to use 2 not 0 i get the CC's output the the console. This means I can't wire them directly up to OSMFCCLib, though, without some manipulation.

Stay tuned for more.

jlacivita · 2015-07-16T21:06:51Z

yay! there's no problem with the flashls TS, PES or NAL parsing. Thanks to mangui's notes above I was able to confirm the packets are in the file, and parsed by flashls in the correct order using a hex editor and some logging. Once I had that confidence, i discovered a bug in my forrest of logging statements where I wasn't decoding the cc_data bytes properly. From there, I cleaned up my FLVTag sorting code, and I've got OSMFCC working with about 90% accuracy.

The remaining problem seems to be in the order the onCaptionInfo events are dispatched. I'm not sure if it's in my caption sorting code or in my new FLVTag() creation, or if it's something subtle in flashls's code for dispatching METADATA tags.

What I do know is that if i reload my page, i see onCaptionInfo events in the wrong order once in a while on the OSMF side, even though they're in the correct order on the flashls parsing side in TSDemuxer. This happens without changing any of my code, just re-running the page.

I think there might be a race condition somewhere in the flashls code that reads FLVTags with the METADATA type and inserts them into the HLSNetStream.

I've committed my latest, so you should be able to see both my forrest of log messages (we can delete this soon, i think!) as well as OSMFCCLib doing it's job (for the most part)

@mangui: i'm ready for your next clue... how do I confirm FLVTag.METADATA messages are going out in the correct order?

jlacivita · 2015-07-21T00:05:08Z

Okay, i've got debug output inside of HLSNetStream (it's not committed, since it's a bit too much logging) that shows appendTags is being called with the FLVTags in the wrong order sometimes (even though they're inserted via parsing in the correct order). Furthermore, it's not always the same tag that is out-of-order, so that rules out any problems with the OSMFCCLib getting mixed up.

Something between TSDemuxer and HLSNetStream is mixing up the FLVTag order for METADATA tags (and maybe others).

mangui · 2015-07-21T07:54:23Z

@jlacivita interesting. do you have a debug log exhibiting the issue to share ?

jlacivita · 2015-07-21T19:33:32Z

@mangui. while preparing the log, i noticed that I've been comparing just the first bitrate's parsing, with the actual playback of a different bitrate.

It turns out that the error does, indeed happen at the parsing level, not the appending level. It's caused by the fact that these messages aren't always in the correct order, and it's up to the HLS client to sort them. I'm sorting them inside the parseAVCPES function, but that grabs a batch of them each time (via _parseTimer) so they're only locally sorted within the batches.

I think they need to be sorted at the Fragment level, so i'll try that now. Log attached, where you can see what i'm saying.

jlacivita · 2015-07-21T19:35:02Z

http://demo.theplatform.com/jeremy/tagorder.log

jlacivita · 2015-07-21T20:20:14Z

Okay that fixed it. i've seen captions work 3 times in a row now, with no mixed up characters.

I'm going to commit to my branch so you can see the changes i've made. I'll leave my debugging code in for now.

it works!

jlacivita · 2015-07-21T20:28:00Z

Commited to my branch. 608/708 Captions are working!!

maxaf · 2015-07-21T23:53:04Z

@jlacivita Bravo! Did you work from any publicly accessible documentation? If so, please share it here. Also, I don't fully understand why live is any different from VOD as far as captions are concerned. Could you elaborate on this?

jlacivita · 2015-07-22T00:01:02Z

I mainly used Wikipedia. MPEG2 H264 streams store 708 captions as described here:
https://en.wikipedia.org/wiki/CEA-708

Once you get the two bytes out of the 708, you can follow the 608 character logic for interpreting them, but that's on the Decoder side, which I haven't really implemented aside from logging out the ASCII characters for debugging (OSMFCCLib is a full featured decoder option):
https://en.wikipedia.org/wiki/EIA-608#Characters

@mangui was also a valuable source of HLS parsing information.

The only reason I backed of from dealing with live is that it always adds extra use cases to any only video development. While I was still trying to understand how it worked, I didn't want live mixed into my head :)

I do think it should work the same way for both live and vod, now that it's working.

maxaf · 2015-07-22T17:12:22Z

I think the difference is that our live streams have plain 608 captions, while the VOD one you worked on has 708. I have yet to see 708 captions in a live stream, but I only have my own streams as specimens.

jlacivita · 2015-07-24T19:03:18Z

@maxaf: who's encoding your live streams? I see they're being cached via akamai hdnetwork, so i'm assuming you're using someone pretty standard for encoding?

Most of the HDS live streams that i've seen use 708 captions. I wonder why HLS would use 608?

I grabbed a single fragment of your stream and didn't see evidence of 608 or 708 in the binary.

If you could get me any details, that would be very helpful

maxaf · 2015-07-27T14:46:53Z

@jlacivita We're using encoders from Cisco's AnyRes Live range, but data sheets don't confirm specifically which captions are being output by the encoder.

jlacivita · 2015-09-02T19:35:36Z

@maxaf: i've confirmed that some live streams work with my changes and some do not. Still trying to track down what the difference is, but I'm guessing it's your theory.

jlacivita · 2015-09-02T19:37:07Z

@mangui now that i've seen 708 working in live streams, i do feel it's time to pull this change into an official flashls branch... What's your opinion?

mangui · 2015-09-03T08:39:57Z

@jlacivita yes it would make sense to merge, but I looked at the diff and it still contains a bunch of debug statement.
IMHO the PR should only contain changes to TSDemuxer and eventually StreamBuffer (if PTS sorting is really needed)

jlacivita · 2015-09-03T21:08:47Z

Okay, all of the debug is removed. The changes are only in TSDemuxer, and StreamBuffer. PTS sorting is definitely needed, although I suppose we could address your concern here with two things:

remove sorting form StreamBuffer
add sorting to the OSMF plug-in code
while we're doing Flash Version 9 #2, you could make a native HLS event for the SEI packets, and transform them into onCaptionInfo events in the OSMF layer as well, since it's a convention I adopted from OSMF.

I still think sorting belongs somewhere in the library, since captions are useful if they're not in the correct order.

jlacivita · 2015-09-03T21:10:25Z

And honestly, #2 seems hard to do w/out creating a whole system for scanning ahead for SEI tags, sorting them, and then dispatching them based on timecode... I really think the sorting belongs in StreamBuffer, so every implementation (OSMF, etc) can benefit from it

mangui · 2015-09-04T08:23:12Z

thanks, looks better !
just curious, do you have a stream available with unsorted SEI tags ?

jlacivita · 2015-09-09T00:13:21Z

most streams i've seen have them unsorted. I just sent you some via email.

mangui · 2015-09-14T09:25:51Z

tks, I am quite busy ATM, will try to check later this week

Conflicts: .gitignore bin/debug/flashls.swc bin/debug/flashlsChromeless.swf bin/debug/flashlsFlowPlayer.swf bin/debug/flashlsOSMF.swc bin/debug/flashlsOSMF.swf bin/release/flashls.swc bin/release/flashlsChromeless.swf bin/release/flashlsFlowPlayer.swf bin/release/flashlsOSMF.swc bin/release/flashlsOSMF.swf

mangui · 2015-09-17T07:41:39Z

tks for taking into account the suggested changes (jlacivita@ee5ed3e)
the issue is that jlacivita@1e4e51e is reverting some upstream changes.
in the end the only expected changes should be in TSDemuxer.as and StreamBuffer.as

Conflicts: .gitignore bin/debug/flashls.swc bin/debug/flashlsChromeless.swf bin/debug/flashlsFlowPlayer.swf bin/debug/flashlsOSMF.swc bin/debug/flashlsOSMF.swf bin/release/flashls.swc bin/release/flashlsChromeless.swf bin/release/flashlsFlowPlayer.swf bin/release/flashlsOSMF.swc bin/release/flashlsOSMF.swf

jlacivita · 2015-09-21T18:26:56Z

Should be fixed

mangui · 2015-09-23T21:10:58Z

that is weird, i still see plenty of unrelated changes
https://github.com/mangui/flashls/pull/334/files
or
dev...jlacivita:dev

Conflicts: bin/debug/flashls.swc bin/debug/flashlsChromeless.swf bin/debug/flashlsFlowPlayer.swf bin/debug/flashlsOSMF.swc bin/debug/flashlsOSMF.swf bin/release/flashls.swc bin/release/flashlsChromeless.swf bin/release/flashlsFlowPlayer.swf bin/release/flashlsOSMF.swc bin/release/flashlsOSMF.swf

jlacivita · 2015-09-24T18:00:18Z

okay, i'm learning how to use git here, so thanks for bearing with me. should be down to just those files (TSDemuxer and StreamBuffer)

adding cleaned up SEI frame-type 6 logic for 608/708 captions

mangui · 2015-09-27T20:34:05Z

thanks @jlacivita !

zuzzurro · 2015-11-05T13:58:14Z

I'm very interested in this feature. Would it be possible to recap what's now available (either in the dev or master branches)? The discusion above is very interesting but it's difficult to follow to a conclusion. Also the test video from @jlacivita (http://www.cpcweb.com/webcasts/hls/cpcdemo.m3u8) seems to have disappeared in the meantime..
Questions:

608 vs 708
Live vs VOD
Enabled out of the box or necessary integrating external libraries and/or additional code? (OSMFCCLib )

Basically, if I have a live or live2vod stream, and I paste it in one of the demo players, am i supposed to see captions (or at least the captions button) or not. If not, what should I do in order to enable it for live streams?

rokclimb15 · 2015-11-06T02:37:41Z

This is 708 for live, but it doesn't work end-to-end out of the box yet as
of my last test. This parses the CC data from the TS packets, but there
still needs to be some work done to align the FLV tags with timing and call
onCaptionInfo or other standard hook for the player to receive the data and
display. VOD CC is already possible via WebVTT sidechannel load.

On Thu, Nov 5, 2015 at 8:58 AM, zuzzurro [email protected] wrote:

I'm very interested in this feature. Would it be possible to recap what's
now available (either in the dev or master branches)? The discusion above
is very interesting but it's difficult to follow to a conclusion. Also the
test video from @jlacivita https://github.com/jlacivita (
http://www.cpcweb.com/webcasts/hls/cpcdemo.m3u8) seems to have
disappeared in the meantime..
Questions:

608 vs 708

Live vs VOD

Enabled out of the box or necessary integrating external libraries
and/or additional code? (OSMFCCLib )

Basically, if I have a live or live2vod stream, and I paste it in one of
the demo players, am i supposed to see captions (or at least the captions
button) or not. If not, what should I do in order to enable it for live
streams?

—
Reply to this email directly or view it on GitHub
#334 (comment).

zuzzurro · 2015-12-03T10:08:28Z

I think it would just be great if @jlacivita could add a little bit of documentation about the integration of his code with the higher levels, like an OSMF player using OSMFCCLib, that I think he used during his development.

Right now implementing CC support and contributing back is more difficult than it could be.

zuzzurro · 2015-12-03T16:19:10Z

I keep following up myself here, but that's probably ok. I was finally able to find out at least the first part of my issue, after having added some debugging logs to the CC code.
I had been testing it with created with one encoder brand, and it would always fail at the "payload_type == 4" test (the type actually always being 0 or 1 instead.

When I switched to another video generated with a different encoder, bingo! it started working and it went all the way down to the _tags.push(tag); line.

Now I have two tasks: 1) figure out what's the difference in the encoder configuration and 2) figure out how to use OSMFCClib..

lvaldivia · 2016-03-10T21:10:15Z

Is there any example to test this funcionality?

adding cleaned up SEI frame-type 6 logic for 608/708 captions

037fca7

This was referenced Jul 10, 2015

Integrate Closed Caption decoding from @jlacivita's repository. #330

Closed

CEA-608 closed captions #11

Open

Cleaned up more code, got captions logging properly

2f9364a

Moved CC FLVTag sorting to StreamBuffer

5593e13

it works!

Merge remote-tracking branch 'mangui/dev' into dev

90ee567

jlacivita added 5 commits September 3, 2015 13:57

Cleaning up debug code

cad6356

fixing gitignore

fa8798d

gitignore

e411460

removing isCC boolean

129e6c2

Removing more debug

a0516cf

jlacivita added 2 commits September 3, 2015 14:06

whitespace cleanup

d25406a

removing extra dir

f9fcb2a

jlacivita added 2 commits September 15, 2015 15:08

using PTS for DTS, and remove compareTagsPTS

ee5ed3e

jlacivita added 7 commits September 24, 2015 10:11

html diffs

e7f4dc4

fixing whitespace diffs

6f98fa9

ignore and API.md

72aca46

gitignore

3e7fbb5

more gitignore

26274a8

binaries

c0df90f

mangui added a commit that referenced this pull request Sep 27, 2015

Merge pull request #334 from jlacivita/dev

053f713

adding cleaned up SEI frame-type 6 logic for 608/708 captions

mangui merged commit 053f713 into mangui:dev Sep 27, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding cleaned up SEI frame-type 6 logic for 608/708 captions #334

adding cleaned up SEI frame-type 6 logic for 608/708 captions #334

jlacivita commented Jul 10, 2015

maxaf commented Jul 15, 2015

mangui commented Jul 15, 2015

jlacivita commented Jul 15, 2015

jlacivita commented Jul 15, 2015

jlacivita commented Jul 16, 2015

jlacivita commented Jul 21, 2015

mangui commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

maxaf commented Jul 21, 2015

jlacivita commented Jul 22, 2015

maxaf commented Jul 22, 2015

jlacivita commented Jul 24, 2015

maxaf commented Jul 27, 2015

jlacivita commented Sep 2, 2015

jlacivita commented Sep 2, 2015

mangui commented Sep 3, 2015

jlacivita commented Sep 3, 2015

jlacivita commented Sep 3, 2015

mangui commented Sep 4, 2015

jlacivita commented Sep 9, 2015

mangui commented Sep 14, 2015

mangui commented Sep 17, 2015

jlacivita commented Sep 21, 2015

mangui commented Sep 23, 2015

jlacivita commented Sep 24, 2015

mangui commented Sep 27, 2015

zuzzurro commented Nov 5, 2015

rokclimb15 commented Nov 6, 2015

zuzzurro commented Dec 3, 2015

zuzzurro commented Dec 3, 2015

lvaldivia commented Mar 10, 2016

adding cleaned up SEI frame-type 6 logic for 608/708 captions #334

adding cleaned up SEI frame-type 6 logic for 608/708 captions #334

Conversation

jlacivita commented Jul 10, 2015

maxaf commented Jul 15, 2015

mangui commented Jul 15, 2015

jlacivita commented Jul 15, 2015

jlacivita commented Jul 15, 2015

jlacivita commented Jul 16, 2015

jlacivita commented Jul 21, 2015

mangui commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

jlacivita commented Jul 21, 2015

maxaf commented Jul 21, 2015

jlacivita commented Jul 22, 2015

maxaf commented Jul 22, 2015

jlacivita commented Jul 24, 2015

maxaf commented Jul 27, 2015

jlacivita commented Sep 2, 2015

jlacivita commented Sep 2, 2015

mangui commented Sep 3, 2015

jlacivita commented Sep 3, 2015

jlacivita commented Sep 3, 2015

mangui commented Sep 4, 2015

jlacivita commented Sep 9, 2015

mangui commented Sep 14, 2015

mangui commented Sep 17, 2015

jlacivita commented Sep 21, 2015

mangui commented Sep 23, 2015

jlacivita commented Sep 24, 2015

mangui commented Sep 27, 2015

zuzzurro commented Nov 5, 2015

rokclimb15 commented Nov 6, 2015

zuzzurro commented Dec 3, 2015

zuzzurro commented Dec 3, 2015

lvaldivia commented Mar 10, 2016