Integrate Closed Caption decoding from @jlacivita's repository.

[gyachuk]

Added new CLOSED_CAPTION_INFO HLSEvent.
Added dispatch of this new event into HLSNetStream.

[kfeinUI]

@gyachuk Looks like you left a good amount of commented code in there. Could you clean that up?

How has your testing experience been with this? Have you tested it against Adobe's OSMFCCLib.swc captioning library? Install the Adobe Media Server trial and find it in with the samples. Closed source, but easy to integrate and does the job painlessly.
https://helpx.adobe.com/adobe-media-server/dev/configure-closed-captioning.html

[gyachuk]

Hey Kevin.
Please don't forget that this isn't my code at all. This is from @jlacivita. It looked like he did initial development against a much older version of @mangui's repo. I simply made the equivalent changes at the current head of the repo, and @mangui asked me for this simpler Pull Request. I really didn't want to change any of @jlacivita's original code.
And thanks for the link. I've already gone through it and am not really sure how to use it. Since OSMFCCLib is a SWC, it has to be used by ActionScript, but I'm using mediaelement as the host system for flashls. mediaelement is all in javascript, so I'm thinking I either have to put the CC handling inside flashls, or write some sort of AS3 wrapper for it.
One thing I'm wondering is how to connect flashls with OSMFCCDecoder. It looks as if you have to set OSMFCCDecoder.mediaPlayer, but there doesn't seem to be anything in flashls that implements MediaPlayer (the required type). I'm figuring I have to somehow mess with HLSLoadFromDocumentElement.as. Any words of wisdom on that?
Greg

 On Monday, July 6, 2015 4:15 PM, Kevin Feinberg <[email protected]> wrote:

@gyachuk Looks like you left a good amount of commented code in there. Could you clean that up?How has your testing experience been with this? Have you tested it against Adobe's OSMFCCLib.swc captioning library? Install the Adobe Media Server trial and find it in with the samples. Closed source, but easy to integrate and does the job painlessly.
https://helpx.adobe.com/adobe-media-server/dev/configure-closed-captioning.html—
Reply to this email directly or view it on GitHub.

[mangui]

dumping parsed NALUs type is already done in Nalu.as, this is duplicated and useless

[mangui]

@gyachuk as u can see, a bunch of cleanup would be needed before considering merging this code

[kfeinUI]

@gyachuk: I was wondering the extent to which you've tested this. Since @jlacivita left off with some unresolved issues, it's unclear if they remain or if they have been resolved as a side effect of changes made to flashls since that time.

OSMFCCLib is provided by Adobe with the assumption that you integrate it into your OSMF-based player build. Don't know why they wouldn't have just added such useful functionality straight into OSMF instead of burying it in the samples of a commercial product, but it's at least easy to work with. You point it to the media player element and wire up controls. The player I'm using is close to stock Strobe which is the reference player packaged with OSMF. In my usage, I created a simple wrapper so that it may function as a standalone plugin (like flashlsOSMF) with the configurations/controls exposed to JavaScript through the ExternalInterface.

If you check in an updated build, I'd be happy to give it some testing.

[gyachuk]

Hey Kevin. My testing has consisted solely of verifying that Base64 encoded onCaptionInfo data that is presented to Javascript, is exactly the same as what the code in @jlacivita's repo does.
I'm still struggling with putting OSMFCCDecoder together with flashs. I've been working with ChromelessPlayer.as and have managed to get it to create an OSMFCCDecoder, but am not sure what to do with it next. Sounds like I should be working with flashlsOSMF instead, right? Any chance you can share your wrapper? Would I need your version of the Strobe player too?
I'll be checking in the cleanups suggested by @mangui later today.
Greg

[gyachuk]

@mangui, I'm happy to clean up that code. My idea was to first present @jlacivita's code with as few changes as possible. I'll update the code a bit later today.

[mangui]

great. thanks

[kfeinUI]

@gyachuk, the essentials on OSMFCCDecoder are:

• mediaPlayer = [your media player instance]
• mediaContainer = [your media player view area which can be used to render]
• type = CCType.CEA708 (it's the only one)
• service = CEA708Service.CC1 (CC1-CC4 available, might be wired up to controls if you need it).
• enabled = true/false to toggle captions (would wire this up to your controls)

That's really all there is to it. The other options are just for presentational customization.

Will look into sharing my plugin. I use Flashls as an external plugin to Strobe, so it would be quite easy for me to drop in a new build to my test player.

[gyachuk]

OK, I've got my changes committed. Excuse me for being a bit of a newb at this, but I assume I have to create a new PR? If so, what happens to the old one? Thanks.

[kfeinUI]

Just check it in to the same branch and the changes will be picked up.
Github's PR process is by branch, so you typically clone and create
branches of your clone for each feature or issue.
On Jul 7, 2015 5:26 PM, "gyachuk" [email protected] wrote:

OK, I've got my changes committed. Excuse me for being a bit of a newb at
this, but I assume I have to create a new PR? If so, what happens to the
old one? Thanks.

—
Reply to this email directly or view it on GitHub
#330 (comment).

[jlacivita]

Hi guys,

my code was not properly connecting to adobes osmfcc lib.

the caption were about 80% there but it seems like some of the packets are dropped.

still debugging why.

I'm I the road now but I can cleanup my code and evaluate @mangui's feedback for any insights later this week. Glad there's some interest here. I'll update my fork and we can all work on it.

[gyachuk]

@kfeinUI, seems I'd committed my changes but hadn't pushed them. I see them in the PR now. Thanks.

[gyachuk]

Hey @jlacivita. Thanks for the great work so far. Any chance you can share the code you're using to connect to OSMFCCDecoder? (p.s., hope you don't mind the integration I did, so far)

[kfeinUI]

@ jlacivita, what's your player setup?

[kfeinUI]

Probably not too much help, but here's the AMS captioning doc if you haven't already found it:
https://helpx.adobe.com/adobe-media-server/dev/configure-closed-captioning.html#support_for_osmf

OSMFCCLib is, not surprisingly, designed for integration with an OSMF-based player. The player bindings are classes from OSMF. Won't work with the Chromeless player.

[jlacivita]

Hi guys. So here's a summary of what i've done:

1. detect frame type of 6 in TSDemuxer
2. when frame type is 6, confirm it's actually 608 captions
3. if so then extract the bytes typically sent in OSMF onCaptionInfo calls (it's not the entire frame data)
4. create an onCaptionInfo message (the defacto OSMF standard for 608 captions) with a payload that matches de-facto object { "type": "708", "data": bytes}
5. spit out a bunch of console messages to compare what I'm expecting to what OSMFCCLib displays

currently i'm getting almost recognizable english out of both 4 & 5, but characters are dropped for some reason, which i'm debugging now.

[kfeinUI]

Cool. Thanks for the update.

Let us know if there's anything we can do to help.

On Fri, Jul 10, 2015 at 4:14 PM, Jeremy LaCivita [email protected]
wrote:

Hi guys. So here's a summary of what i've done:

1. detect frame type of 6 in TSDemuxer
2. when frame type is 6, confirm it's actually 608 captions
3. if so then extract the bytes typically sent in OSMF onCaptionInfo calls
   (it's not the entire frame data)
4. create an onCaptionInfo message (the defacto OSMF standard for 608
   captions) with a payload that matches de-facto object { "type": "708",
   "data": bytes}
5. spit out a bunch of console messages to compare what I'm expecting to
   what OSMFCCLib displays

currently i'm getting almost recognizable english out of both 4 & 5, but
characters are dropped for some reason, which i'm debugging now.

—
Reply to this email directly or view it on GitHub
#330 (comment).

[bwalls]

The characters are all there in Safari? You’re sure the test stream is properly captioned? I’ve seen chopped up captions from an encoder being set to drop the frame rate. That can be really hard to debug if it’s an adaptive stream, and the lower rates through away some of the letters in the captions. Not a helpful feature...

On Jul 10, 2015, at 3:14 PM, Jeremy LaCivita [email protected] wrote:

Hi guys. So here's a summary of what i've done:

1. detect frame type of 6 in TSDemuxer
2. when frame type is 6, confirm it's actually 608 captions
3. if so then extract the bytes typically sent in OSMF onCaptionInfo calls (it's not the entire frame data)
4. create an onCaptionInfo message (the defacto OSMF standard for 608 captions) with a payload that matches de-facto object { "type": "708", "data": bytes}
5. spit out a bunch of console messages to compare what I'm expecting to what OSMFCCLib displays

currently i'm getting almost recognizable english out of both 4 & 5, but characters are dropped for some reason, which i'm debugging now.

—
Reply to this email directly or view it on GitHub #330 (comment).

[jlacivita]

Yes, the characters all work in safari just fine. I've even made copies of the M3U8 that have only 1 bitrate and one fragment, and it works in Safari, but not in my fork of flashls

[jlacivita]

Hi guys. I spent some time learning how to use GitHub, and have an up-to-date fork of flashls with my changes, cleaned up according to mangui's suggestions, but still not working (see my comment 3 hours ago).

I've made a new pull request:
#334

So we can try and solve this together, but I believe the problem is either:

The order the FLVTags are sorted. I've got some code to splice them in based on PTS values that improved the captions compared to just pushing them on.

or

Something upstream in flashls preventing all of the frame type 6 frames from being surfaced.

[gyachuk]

@jlacivita, thanks for updating yours. I'm getting an error when I ./build.sh, cause it seems that LevelController isn't defined.

[jlacivita]

@gyachuk LevelController is in my fork. Try deleting your clone of my repro and re-cloning it.

[gyachuk]

Thanks for the suggestion @jlacivita. I'm now able to build, but I think I'm still missing something basic. In your repo, I only see a single commit from you. It's in the dev branch, and it is only to TSDemuxer.as. I can see the logging for "the_cc_data" in my browser console, so I know it is being collected by TSDemuxer, but I don't see how it is getting out from the flashls code. I'm using index.html in the ChromelessPlayer example. I don't see anywhere that you're wiring up a MediaPlayer, or the OSMFCCDecoder.

Are you decoding within the flashls code? Or are you somehow getting the sei_data out and into something else?

Would love to help on this, but I don't seem to quite get what you're doing. Thanks.

[jlacivita]

@gyachuk I'm using my own custom OSMF MediaPlayer and the OSMFCClib, not the chrome less player sample distributed with flashls.

Any NetStream based player can consume the onCaptionInfo events that my code inserts into the stream, OSMF just makes it easier, especially if you add the OSMFCClib to render them.

Just make a new SWF with a MediaPlayer and an OSMFCCDecoder and the flashls plugin and you should see the garbled rendering.

[mangui]

outdated PR, closing it