HLS - new RollingSampleBuffer NullPointerException in 1.5.12 (Not present in 1.5.6)

[brAzzi64]

We recently upgraded from ExoPlayer r1.5.6 to r1.5.12, and have been seeing NullPointerExceptions from ExoPlayer.

We were also able to reproduce it with the Demo App, by just adding a Sample pointing to our media file (an HLS playlist).

The NPE always happens when dereferencing an instance of DefaultTrackOutput, but when calling various methods of that instance. It happens with several media files, but in a very specific position for each, by just letting it play from the beginning. However, it only happens sporadically; in my tests frequency was 1 out of 10 for a particular media file.

Testing device is a Nexus 6P, with 7.1.1.

Our media URLs are not public, so I'll be sending it to [email protected].

Finally, here's a sample exception:

02-07 16:04:28.580 1642-2214/? E/ExoPlayerImplInternal: Internal runtime error.
                                                        java.lang.NullPointerException: Attempt to read from field 'com.google.android.exoplayer.extractor.RollingSampleBuffer com.google.android.exoplayer.extractor.DefaultTrackOutput.rollingBuffer' on a null object reference
                                                            at com.google.android.exoplayer.extractor.DefaultTrackOutput.configureSpliceTo(DefaultTrackOutput.java:188)
                                                            at com.google.android.exoplayer.hls.HlsExtractorWrapper.configureSpliceTo(HlsExtractorWrapper.java:157)
                                                            at com.google.android.exoplayer.hls.HlsSampleSource.readData(HlsSampleSource.java:315)
                                                            at com.google.android.exoplayer.SampleSourceTrackRenderer.readSource(SampleSourceTrackRenderer.java:208)
                                                            at com.google.android.exoplayer.MediaCodecTrackRenderer.feedInputBuffer(MediaCodecTrackRenderer.java:636)
                                                            at com.google.android.exoplayer.MediaCodecTrackRenderer.doSomeWork(MediaCodecTrackRenderer.java:529)
                                                            at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:128)
                                                            at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:432)
                                                            at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:211)
                                                            at android.os.Handler.dispatchMessage(Handler.java:98)
                                                            at android.os.Looper.loop(Looper.java:154)
                                                            at android.os.HandlerThread.run(HandlerThread.java:61)
                                                            at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)

bugreport-N4F26I-2017-02-07-16-10-46.zip

[AquilesCanta]

It's giving me 403. Can you send another one?

[brAzzi64]

Sent to [email protected].

Thanks!

[AquilesCanta]

Hello, I was not able to reproduce. I have tried version 2.2.0, 1.5.14 and 1.5.12, so I assume this particular stream does not cause the failure. Could you provide a link that reproduces the issue in the DemoApp in either version 2.2.0 or 1.5.14? I don't remember whether there's a bugfix for this particular issue in between, but there might be, in which case the only solution would be to update or manually backport the fix.

Once you have a link, please provide the reproduction steps via e-mail. Thanks!

[brAzzi64]

I tried this again just now, and I can confirm that this happens on the demo app form r1.5.12, with the media that I sent (URLs might be different, but the media I'm sending is always be the same, which causes the issue at time 0:09 seconds). Please note however that it took me 8 runs to actually hit the issue.

This is how I reproduced it:

1. Tap on the right video under the HLS menu
2. Wait for the video to hit 12 secs
   • If you run into it, the video will freeze at 9 secs you'll get the exception
   • If it doesn't, press back to go back to the menu, and tap the video again

Let me know if there's anything else I can provide.

[AquilesCanta]

Heck, it has just expired. I think I know what the issue is. You can check this yourself. It might be the case that some of the variants include audio only? If this is the case, there is no more need for debugging, I think.

Adapting to audio only variants is not supported currently. I need to discuss with my team whether it's going to be added to the roadmap. In any case, it's most likely going to be added to V2 only, so keep that in mind. But I highly doubt it.

Regarding the fix: Either add a CODECS attribute to the EXT-X-STREAM-INFs stating the actual codecs being used by the variants or directly remove those tags from the playlist. The CODECS attribute is the way to go, it has been around this HLS version 1, and marked as SHOULD later on. I don't really see a good reason not to add it. So variants with video will have something like CODECS="avc...,mp4a..." and audio only something like CODECS="mp4a..." without the avc part. Please confirm here if my supposition is correct (was about to do it just when the url expired), and if so, feel free to close the issue. If not, we can try again tomorrow.

[brAzzi64]

I did some digging through the .m3u file references, and didn't land on a playable video, so I couldn't figure it out myself; I'll read up a bit on the HLS spec. I'll also reach out to the appropriate team on my side and ask if that could be the case.

On the meantime, I'm sending you an updated URL in case you can find out about this before me.

Thanks!

[AquilesCanta]

After looking into the transport streams I can confirm that there are at least two audio-only variants. So my comment still applies. Adding the codecs information is the best you can do right now. You can track #545 if you are interested in this feature.

[brAzzi64]

Awesome, thanks!

[brAzzi64]

Hi Aquiles. After some more digging on our side, we discarded the presence of audio-only streams being the issue. In the provided playlist in particular, there's no audio only streams. They're all audio+video.

Ran several tests for a new, single video, by only querying a single resolution at a time (generating playlists of a single stream) and found that for any variant of the video, scrubbing anywhere in the 10 sec to 20 sec range will consistently cause the failure on ExoPlayer. Anywhere outside of that range, works fine.

Sending the original URL to [email protected], along with this URL modified to return a playlist with a single video stream variant.

Is it possible to re-open this issue, or should we open a new one?

Thanks!

[AquilesCanta]

Both links give me 403. Can you fix this?

[brAzzi64]

Yes, sent you updated links to [email protected]. Thanks!

[AquilesCanta]

Every second chunk of the variants does not declare video (but does contain the actual payload, which explains why it works when the player can reuse the extractor).

[brAzzi64]

Hi Aquiles, thanks for looking into this again. I'll pass this comment to the video-encoding people here, and see what we can do about it.

The thing is that we have several videos in production causing this issue, videos that used to play fine in with earlier ExoPlayer versions. I tried all the version from 1.5.6 to 1.5.12, and pinpointed it to 1.5.11; from this version on, failures in these videos will occur consistently.

Was there any change that would cause ExoPlayer to be less resilient to problems in the video format? Do you know if there's a patch that could be applied to avoid this issue at least temporarily, while we work on fixing the video? Some workaround for this would be extremely helpful, or at least pointers to look for one.

Sent updated links to [email protected].

Thanks much!

[brAzzi64]

After some more testing, found that this happens consistently only for r1.5.11 & r1.5.12; however, it still happens 1 out of 5 times on the latest, r1.5.14.

In particular, these two Exceptions happen often in that version, always when seeking to anywhere in the range of 10-20 secs of that video:

Internal runtime error.
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2
    at com.google.android.exoplayer.hls.HlsExtractorWrapper.getMediaFormat(HlsExtractorWrapper.java:185)
    at com.google.android.exoplayer.hls.HlsSampleSource.readData(HlsSampleSource.java:329)
    at com.google.android.exoplayer.SampleSourceTrackRenderer.readSource(SampleSourceTrackRenderer.java:208)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.feedInputBuffer(MediaCodecTrackRenderer.java:636)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.doSomeWork(MediaCodecTrackRenderer.java:529)
    at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:128)
    at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:432)
    at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:211)
    at android.os.Handler.dispatchMessage(Handler.java:98)
    at android.os.Looper.loop(Looper.java:154)
    at android.os.HandlerThread.run(HandlerThread.java:61)
    at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)

Internal runtime error.
java.lang.NullPointerException: Attempt to invoke virtual method 'void com.google.android.exoplayer.extractor.DefaultTrackOutput.discardUntil(long)' on a null object reference
    at com.google.android.exoplayer.hls.HlsExtractorWrapper.discardUntil(HlsExtractorWrapper.java:212)
    at com.google.android.exoplayer.hls.HlsSampleSource.discardSamplesForDisabledTracks(HlsSampleSource.java:644)
    at com.google.android.exoplayer.hls.HlsSampleSource.continueBuffering(HlsSampleSource.java:260)
    at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:126)
    at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:432)
    at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:211)
    at android.os.Handler.dispatchMessage(Handler.java:98)
    at android.os.Looper.loop(Looper.java:154)
    at android.os.HandlerThread.run(HandlerThread.java:61)
    at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)

I'll be sending additional links (they only last 4hs and then expire) at 0:00am PST, in case you can take a look at them during the morning in London time.

Thanks!

[AquilesCanta]

For what it's worth, it is somewhat harder to reproduce in V2, and if we were to introduce any modification, it would be in that version.

I am sorry to hear that some streams used to work and no longer do. We are usually flexible about this, if the fix is simple and relatively sensible. But in this case, I can see no simple way to patch Exoplayer. Even less using assumptions that make sense in a majority of cases. It is impracticable to maintain a correct implementation if we patch it to keep support of all non-spec media. In the same way, we cannot provide a well defined behavior for all incorrect media. if it works, it does not guarantee it will keep working. The ground truth is always the spec.

[brAzzi64]

I understand, the explanation makes sense. We want to work on getting our video up-to-spec.

For some context, these links are being served by Edgecast. We found that when served through Akamai we don't see this issue at all, so there might be something about the way Edgecast generates the playlist/chunking that's making it fail.

Now, it would help us debug the issue on our side if you were able to explain "every second chunk of the variants does not declare video, but it does contain payload" a bit further.

What exactly is the spec that our playlist/video is not complying to? What do you mean by "does not declare video"? Are you referring to the playlist containing the chunks? Or specifically to chunk #2 of that playlist, the .ts file?

The video production team mentioned one possibility could be chunks being split in non-keyframe frames, as the cause of the issue. Is this what you're referring to?

Any hints will be greatly appreciated.

Thanks!

[AquilesCanta]

I'll do my best:

What exactly is the spec that our playlist/video is not complying to?

The violated spec is not HLS itself, but the Transport Stream spec (ISO/IEC 13818-1).

The Program Map Table [...] specifies, among other information, which PIDs, and therefore
which elementary streams are associated to form each program.

What do you mean by "does not declare video"?

So this is what the first chunks look like:

        Stream_type loop: 

            Stream_type: 27 (0x1b)  [= AVC video stream as defined in ITU-T Rec. H.264 | ISO/IEC 14496-10 Video]
            reserved_1: 7 (0x07)
            Elementary_PID: 257 (0x0101)
            reserved_2: 15 (0x0f)
            ES_info_length: 0 (0x0000)


            Stream_type: 15 (0x0f)  [= ISO/IEC 13818-7 Audio with ADTS transport sytax]
            reserved_1: 7 (0x07)
            Elementary_PID: 258 (0x0102)
            reserved_2: 15 (0x0f)
            ES_info_length: 0 (0x0000)

This is okay, there is one PAT, one PMT and the PMT declares, as you can see, an audio and a video elementary stream. This was the output of dvbsnoop -tssubdecode -s ts -if 456826_00_03_MM30_Exercise_1650.mp4Frag1Num0.ts 256.

Now if we look at dvbsnoop -tssubdecode -s ts -if 456826_00_03_MM30_Exercise_1650.mp4Frag1Num1.ts 256. (note the file is Frag1Num1 now, the second chunk).

 Stream_type loop: 

            Stream_type: 15 (0x0f)  [= ISO/IEC 13818-7 Audio with ADTS transport sytax]
            reserved_1: 7 (0x07)
            Elementary_PID: 258 (0x0102)
            reserved_2: 15 (0x0f)
            ES_info_length: 0 (0x0000)

There is only audio. As a side note, the third chunks are okay again. However, the PID 257 (which corresponds to video in the first chunk) is populated in the second chunk, so I suspect only the PMT is wrong. The media content is actually there.

Are you referring to the playlist containing the chunks?

No, I was referring to the chunks themselves. Playlist looked fine to me, though I was not looking for anything there, so don't take my word.

Or specifically to chunk #2 of that playlist, the .ts file?

Specifically chunk number 2 of every media playlist (I didn't actually check them all, I think). The snippets above were obtained from the 1650000 variant.

The video production team mentioned one possibility could be chunks being split in non-keyframe frames, as the cause of the issue. Is this what you're referring to?

No, that is not enforced by the spec and we do not count on it. Though it's always better, as stated by the spec:

The server SHOULD attempt to divide the source media at points that support 
effective decode of individual Media Segments, e.g. on packet and key frame boundaries.

I hope this answers your questions.

[brAzzi64]

That was extremely helpful Aquiles.

I ran dvbsnoop against the second chunk and could verify the issue, report to our CDN, and the identified the problem.

Thanks again!