Getting stream sample format

[nbonamy]

[REQUIRED] Searched documentation and issues

Looked into the source code and here but could not find the info.

[REQUIRED] Question

I am making good progress on my (ffmpeg based) transcoding engine but I realize that I need to have access to the Sample Format information (sample_fmt in ffmpeg or equivalent in ExoPlayer) of the stream to decode/resample it properly. The field can be shown using ffprobe but I think this info is attached to the stream which is not accessible.

Also the profile would also be neat to get in order to make a difference between DTS and DTS-HD although they are supported by the same codec and have the same mime-type.

[STREAM]
index=2
codec_name=dts
codec_long_name=DCA (DTS Coherent Acoustics)
profile=DTS-HD MA
codec_type=audio
sample_fmt=s32p

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[andrewlewis]

On the extraction side I'm afraid we don't expose the audio bit depth for compressed audio formats (only for uncompressed/PCM audio where it's stored in Format.pcmEncoding).

I'm a bit surprised this information is needed as I think generally audio decoders will either output in an encoding depending on the content (and allow you to query what this is), or allow you to set an output encoding and resample internally, so maybe I have misunderstood what you're trying to do. MediaCodec audio decoders for compressed formats output 16-bit PCM. In the ExoPlayer ffmpeg extension there's a mode you can enable to output 32-bit float PCM.

There are separate MIME types for DTS and DTS-HD, but there's a current limitation where for
the Matroska container we can't currently distinguish these so always assume DTS (see #6225).

[nbonamy]

Thanks. I am expanding the ffmpeg extension so that it can transcode to AC3 (or anything else but given I have a Sonos system, AC3 is what I need) instead of just PCM. It makes a huge difference in terms of sound quality.

Not being an expert, but setting the sample_fmt in this case seems not as simple as in the PCM case (context->request_sample_fmt = outputFloat ? AV_SAMPLE_FMT_FLT : AV_SAMPLE_FMT_S16;).

Maybe my transcoding engine is the culprit but it is based on https://ffmpeg.org/doxygen/4.1/transcode_aac_8c-example.html where basically the sample_fmt and request_sample_fmt fields get populated from the stream data (line 107: error = avcodec_parameters_to_context(avctx, (*input_format_context)->streams[0]->codecpar);) and it gets the stream sample_fmt (codecpar->format, https://ffmpeg.org/doxygen/3.2/structAVCodecParameters.html#abee943e65d98f9763fa6602a356e774f).

Anyway having inspected the MKV headers using mkvtoolnix I realize this information is not even available there so there is probably not much that can be done here. Maybe I could try and write a custom extractor based on FFmpeg and therefore get that information and store it in Format. What do you think?

About the DTS/DTS-HD, I also saw that my video file had A_DTS instead of of A_DTS/LOSSLESS in the header. When I updated it with mkvtoolnix, things went fine so I am surprised by #6225: detection on ExoPlayer is fine but it seems more an issue with the encoder who did not populate the header properly.

Thanks
Nicolas

[andrewlewis]

In this setup, I'm not sure where the format information is missing -- can't you set up the extension always to decode to 16-bit PCM (or float if you prefer, though I think this won't much much difference to quality given that AC-3 decodes to 16-bit PCM anyway), then add in your encoding step to AC-3 after that, where the input format is known? Sounds like a fun/useful project!

[nbonamy]

Yes that was my initial take at it but then trying several different input codecs (DTS, DTS-HD, TrueHD, Dolby Digital Plus...) I could not get a consistent transcoding (crashes, no sound, static...). When I started using sample_fmt as given by ffprobe then only I started to get consistent results. Using AV_SAMPLE_FMT_S16 for instance would work for some files but not for others. Same with AV_SAMPLE_FMT_FLT of course.

To be honest I do not understand why this info is not required when resampling to PCM but is when resampling to AC3: this is beyond my current comprehension 😀

So the question is: is writing a custom extractor tough or "straightforward"?

Thanks
Nicolas

[andrewlewis]

Going on the assumption that the current code in the ffmpeg extension correctly resamples to 16-bit or float PCM (as requested by the Java layer), can't you just keep the resampling step and set up your encoder after that? Then it should always be getting input in one of those two formats. I think it's better to figure out what's going wrong in the ffmpeg wrapping code rather than implementing a custom extractor, which I wouldn't describe as straightforward.

[nbonamy]

OK will look into that as I agree: there must be something. I will also try to auto-select the sample_fmt from the sample_fmts array: there is surely a way to find the "best" one automatically. Will keep you updated!

[andrewlewis]

I'll close this as at this point it is more related to ffmpeg than ExoPlayer, but feel free to post your findings if you discover what's going on before the issue is automatically locked. Thanks.

[nbonamy]

So after more digging:

• AC3 encoder requires fltp (Float Planar, confirmed here https://hydrogenaud.io/index.php/topic,109796.0.html)
• Some decoders work fine when you set their request_sample_fmt to fltp (DTS), others don't (DTS-HD produces silence)
• Some decoders advertise sample_fmts but even selecting the good one automatically does not seem so easy. Others do not advertise any so there is no way to guess (Dolby TrueHD for instance).
• Tried using ac3_fixed codec which uses s16p sample format but with no better results.

So I am stuck and require the codecpar->format to be available. For the moment I can work around that as files are served by a custom UPnP server so I made the server tell my Android TV app what is the sample_fmt to use. But this is a bit hacky and ultimately I may have to write a custom extractor. Or not 😛

Thanks for the discussion @andrewlewis!
Nicolas

[andrewlewis]

I'm still wondering whether it would work to keep a resampling step from whatever the decoder outputs to float planar in between the decoder and AC-3 encoder. It seems that would be the simplest solution.

[nbonamy]

That’s what I am doing. Basically:

• decode providing a request_sample_fmt (and try to get format of encoder (fltp))
• resample to destination forrmat of encoder, number of channels and sample rate
• accumulate packet in a FIFO in case decoder does not provide packets big enough to be encoded
• encode to final format

I understand that you say that I should be able to do it the same way as PCM decoding but I tried every combination and could not get consistent results unless I set request_sample_fmt to the format of the stream and still really do not know why.

My guess is that simply decoding to PCM and then feeding that to AudioSink is simpler than encoding to AC3. I guess also that the FFmpeg transcoding sample would not set request_sample_fmt the same way if it was not needed.

I will do one last thing: tweak the FFmpeg transcoding sample to make it similar to how we think it should work and see if I get something.

[nbonamy]

Another day, another fail 😅

Tried to get the sample_fmt from the decoded AVFrame as suggested here: https://stackoverflow.com/questions/37235264/ffmpeg-avcodec-open2-changed-audio-format-in-avcodeccontext. Seemed like a very good lead. To no avail 😒

[nbonamy]

Adding more suff if anybody gets here 😀

Dumped this info from ffmpeg. There is clearly real simple rules that could be built from this data but this is definitely not enough to cover all sorts of codecs...

Codec	Profile	BitsPerSample	SampleFmt	SampleFmts
aac	LC	0	fltp	fltp
ac3	(null)	0	fltp	fltp
dca	DTS	0	fltp	s16p,s32p,fltp
dca	DTS-HD HRA	0	fltp	s16p,s32p,fltp
dca	DTS-HD MA	16	s16p	s16p,s32p,fltp
dca	DTS-HD MA	24	s32p	s16p,s32p,fltp
eac3	(null)	0	fltp	fltp
truehd	(null)	24	s32	none