Add more separate hardware decoding toggles, support videotoolbox #2809

nyanmisaka · 2020-04-08T17:08:41Z

Changes

Allow users to customize whether to use the hardware decoder in a specific format when using VAAPI or AMF.
Allow hardware decoding to be turned off when transcoding HEVC or VP9 10-Bit videos. (Especially for older GPUs that do not support 10-Bit decoding)
Provide support for VP9/VP8 QSV decoder and VP9/VP8 NVDEC decoder coming in FFmpeg 4.3 in the future.
Continue to fix errors encountered in UTF-16 format SRT subtitles. (see 2503)
Completing some unlisted codecs that should be supported.

Artiume · 2020-04-15T08:45:55Z

I can't remember, but I believe that there were some situations where 10bit h264 isn't hw supported either. Should we expand the 10bit toggle to include it?

I've been working with #2873 I think it might benefit from this PR. The i3 cpu seems to be missing some hw profiles.

nyanmisaka · 2020-04-15T08:57:42Z

I can't remember, but I believe that there were some situations where 10bit h264 isn't hw supported either. Should we expand the 10bit toggle to include it?

I've been working with #2873 I think it might benefit from this PR. The i3 cpu seems to be missing some hw profiles.

Currently, only HEVC/VP9 may have a 10-bit hardware decoder. Therefore, in addition to these two formats, a software decoder should be used when encountering 10-bit video.

https://github.com/jellyfin/jellyfin/blob/a388d5dd72893fa0a4e30472bd1d5fc8854ad51d/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs#L2550#L2556

JustAMan · 2020-04-20T15:24:53Z

I'll review shortly, but otherwise I find this approach a bit wrong. I mean, we should be able to do a few tests on the real HW to determine which ticks work and which don't automatically. So the ticks should be for overriding when stuff works but with some glitches.

That's just me thinking out loud, though, not a blocker for this PR (and not to be addressed here).

JustAMan

I'm not so sure about blanket "enable/disable 10-bit HW support", what if chip can decode 10-bit HEVC but cannot do 10-bit VP9?..

MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs

MediaBrowser.Controller/MediaEncoding/IMediaEncoder.cs

MediaBrowser.MediaEncoding/Encoder/EncoderValidator.cs

MediaBrowser.MediaEncoding/Encoder/MediaEncoder.cs

JustAMan · 2020-04-20T15:52:21Z

@nyanmisaka is there a companion PR for jf-web to add the checkbox and localisation?

nyanmisaka · 2020-04-20T17:42:45Z

@nyanmisaka is there a companion PR for jf-web to add the checkbox and localisation?

jellyfin/jellyfin-web#1046

nyanmisaka · 2020-04-20T17:50:19Z

I'm not so sure about blanket "enable/disable 10-bit HW support", what if chip can decode 10-bit HEVC but cannot do 10-bit VP9?..

This may indeed happen. So I need to add another toggle to distinguish them. Or is there any other good way?

nyanmisaka · 2020-04-20T18:05:17Z

I'll review shortly, but otherwise I find this approach a bit wrong. I mean, we should be able to do a few tests on the real HW to determine which ticks work and which don't automatically. So the ticks should be for overriding when stuff works but with some glitches.

That's just me thinking out loud, though, not a blocker for this PR (and not to be addressed here).

I also have the idea of being able to detect system and HWA capabilities through a utility program. And that's what Emby next door did. They wrote a ffdetect that can export HWA capabilities in json and other formats. Therefore, the ffmpeg command can be generated more accurately, and users do not need to manually tick when using HWA.

JustAMan · 2020-04-20T22:49:36Z

This may indeed happen. So I need to add another toggle to distinguish them. Or is there any other good way?

Maybe just list those as separate toggle-able codecs, like HEVC and HEVC 10-bit?

They wrote a ffdetect that can export HWA capabilities in json and other formats.

Cannot we just call ffmpeg and tell it to transcode some still image to a video using HWA?
Also we can use same still image, encode it in software and try decoding with HWA.

Maybe we can even do without image and just make a pad filter with black 10x10 box or smth.

EraYaN · 2020-05-13T11:49:05Z

You will need the image because you need to check output colors. I've toyed with the idea, but the problem is reported support is not reliable. It was much more complicated you essentially need to try every single one and then do a bitmap compare and have some threshold as to when it's "unusable"

But really the toggles would need to be a matrix almost, one side the APIs and the top side the codecs (including YUV4:2:2 vs YUV4:2:0 etc but also resolution) with an auto detection thing too. I tried to make it but not in C#. It worked but was slow and took a good 15 minutes to complete. But then we can fill the matrix with tick marks of what we think works and then let the user adjust if required. And put warnings and crosses in there too if it doesn't work or if it's broken in some way. The added difficulty of HDR is a whole other ball game. It's basically not supported right now, but for nvenc someone made a stream post processor.

Artiume · 2020-05-25T21:33:02Z

Updated comments are here nyanmisaka#4

I also added a few sections for videotoolbox, I'll add a few more in a bit

This also includes the same fix for #2127 and allows it to be closed.

graphical subtitles can be off-center in some cases

rigtorp · 2020-05-29T00:56:49Z

When using vaapi the command vainfo outputs what codecs are supported. Have we tried --hwaccel auto?

rigtorp · 2020-05-29T01:13:01Z

Kodi has a list of sample files: https://kodi.wiki/view/Samples

I wonder if we could find a set of files in public domain for each codec and profile and just try each to see what is supported on JF startup.

Artiume · 2020-05-29T13:18:49Z

vainfo would be good to parse and store. As for samples, I think @EraYaN and @JustAMan's idea to use a small image/clip and process it to verify operations. Pulling samples online seems nasty, especially in a bandwidth restricted situation.

Artiume · 2020-05-29T14:28:34Z

nyanmisaka#5

Updated to split HEVC/VP9 10bit. Changed video toolbox's decoder to opencl. We need to use the libs from here https://github.com/upsuper/ffmpeg-vdadec and add --enable-opencl to ffmpeg.

crobibero · 2020-06-02T14:35:51Z

MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs

+                        .Append(encodingOptions.VaapiDevice)
+                        .Append(" ");
+                }
+                else if (!isVaapiDecoder && isVaapiEncoder)


isVaapiDecoder is always false here.

Artiume · 2020-06-02T15:01:25Z

Blocked pending web reviews and my PR's being commited

JustAMan · 2020-06-08T11:49:18Z

MediaBrowser.Model/Configuration/EncodingOptions.cs

+        public bool EnableDecodingColorDepth10Hevc { get; set; }
+        public bool EnableDecodingColorDepth10Vp9 { get; set; }


To be more future-proof we might want to nest a structure of bools, so it would become EnableDecodingColorDepth10.CodecName. Could be done later if needed, though.

Artiume · 2020-06-10T11:28:30Z

/azp run

azure-pipelines · 2020-06-10T11:28:34Z

Commenter does not have sufficient privileges for PR 2809 in repo jellyfin/jellyfin

Bond-009 · 2020-06-10T11:37:02Z

/azp run

azure-pipelines · 2020-06-10T11:37:12Z

Azure Pipelines successfully started running 1 pipeline(s).

davispuh · 2020-08-29T23:21:37Z

This is really good and worked for me (I disabled 10-Bit hardware decoding for HEVC), but still there should be auto detection and fallback to software.

For me I got playback error in client which doesn't really say much.
Then looking at logs I saw

[2020-08-29 23:43:22.541 +03:00] [INF] [77] MediaBrowser.Api.Playback.Hls.DynamicHlsService: Current HLS implementation doesn't support non-keyframe breaks but one is requested, ignoring that request
[2020-08-29 23:43:22.541 +03:00] [INF] [77] MediaBrowser.Api.Playback.Hls.DynamicHlsService: /usr/bin/ffmpeg -c:v hevc_cuvid -resize 1280x536 -i file:"/mnt/movie.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_nvenc -pix_fmtyuv420p -preset slow -b:v 2552000 -maxrate 2552000 -bufsize 5104000 -profile:v high  -g 72 -keyint_min 72 -sc_threshold 0 -start_at_zero -vsync -1 -codec:a:0 libmp3lame -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -f hls -max_delay 5000000 -hls_time 3 -individual_header_trailer 0 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcoding-temp/transcodes/transcodes/bacfb5d5249b1e91877800fc3202d00d%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcoding-temp/transcodes/transcodes/bacfb5d5249b1e91877800fc3202d00d.m3u8"
[2020-08-29 23:43:22.688 +03:00] [ERR] [19] MediaBrowser.Api.Playback.Hls.DynamicHlsService: FFMpeg exited with code 1
[2020-08-29 23:43:22.762 +03:00] [WRN] [47] MediaBrowser.Api.Playback.Hls.DynamicHlsService: cannot serve "/var/lib/jellyfin/transcoding-temp/transcodes/transcodes/bacfb5d5249b1e91877800fc3202d00d0.ts" as transcoding quit before we got there

Which also is pretty useless, I see that ffmpeg failed. Okay so then I run this command manually to find out that ffmpeg says

[hevc_cuvid @ 0x564beb15d300] Codec hevc_cuvid is not supported.

And only now I know what's the issue. Terrible user experience. I shouldn't have to know about any of this.
Jellyfin should automatically try hardware decoding and if it fails then do in software.
You could cache results so it has to try only once in a while.

cvium · 2020-08-30T19:08:41Z

@davispuh You're not wrong, but don't be an ass about it. Feel free to submit a PR to fix it 🙂

nyanmisaka mentioned this pull request Apr 8, 2020

Add more separate hardware decoding toggles jellyfin/jellyfin-web#1046

Merged

nyanmisaka mentioned this pull request Apr 20, 2020

10.5.3 some subtitles cannot be centered #2840

Closed

nyanmisaka requested a review from a team April 20, 2020 06:22

JustAMan suggested changes Apr 20, 2020

View reviewed changes

JustAMan mentioned this pull request Apr 20, 2020

Hardware decoding untick after playing AVI/MPEG4 files #2528

Closed

JustAMan mentioned this pull request May 13, 2020

V4L2 issues with ffmpeg jellyfin/jellyfin-ffmpeg#29

Closed

anthonylavado added the merge conflict Merge conflicts should be resolved before a merge label May 16, 2020

nyanmisaka requested a review from joshuaboniface as a code owner May 26, 2020 00:47

barronpm force-pushed the hwaccel branch from 62042cf to a388d5d Compare May 26, 2020 00:58

joshuaboniface requested a review from JustAMan May 26, 2020 02:08

nyanmisaka and others added 8 commits May 26, 2020 18:06

add more separate hw decoding toggles

b4b9399

resolve conflicts

c4ba71d

minor changes

161b2a2

probe Main/High 10 more specifically

695f20b

minor changes

5fd3ea8

drop 'force_original_aspect_ratio'

22ef0e3

graphical subtitles can be off-center in some cases

Comments

0eb5791

Update IMediaEncoder.cs

62e47d0

anthonylavado removed the request for review from joshuaboniface May 26, 2020 19:32

crobibero approved these changes Jun 2, 2020

View reviewed changes

Artiume added the blocked Blocked by another pull request label Jun 2, 2020

JustAMan and others added 5 commits June 8, 2020 14:44

Merge remote-tracking branch 'upstream/master' into hwaccel

4999831

Split VP9 HEVC 10bit button. MacOS Decoder fixed to opencl

347689e

fix opencl10bit

d650440

Update EncoderValidator.cs

0476acf

Split HEVC VP9 10bit

b79957d

JustAMan approved these changes Jun 8, 2020

View reviewed changes

Artiume removed the blocked Blocked by another pull request label Jun 8, 2020

Merge branch 'master' into hwaccel

dabb869

anthonylavado merged commit 25f8e59 into jellyfin:master Jun 16, 2020

raouldc mentioned this pull request Aug 1, 2020

Add VideoToolbox to FFmpeg HWA options #3121

Closed

nyanmisaka deleted the hwaccel branch August 3, 2020 05:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more separate hardware decoding toggles, support videotoolbox #2809

Add more separate hardware decoding toggles, support videotoolbox #2809

nyanmisaka commented Apr 8, 2020

Artiume commented Apr 15, 2020

nyanmisaka commented Apr 15, 2020

JustAMan commented Apr 20, 2020

JustAMan left a comment

JustAMan commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

JustAMan commented Apr 20, 2020

EraYaN commented May 13, 2020 •

edited

Loading

Artiume commented May 25, 2020 •

edited

Loading

rigtorp commented May 29, 2020

rigtorp commented May 29, 2020

Artiume commented May 29, 2020

Artiume commented May 29, 2020

crobibero Jun 2, 2020

Artiume commented Jun 2, 2020

JustAMan Jun 8, 2020

Artiume commented Jun 10, 2020

azure-pipelines bot commented Jun 10, 2020

Bond-009 commented Jun 10, 2020

azure-pipelines bot commented Jun 10, 2020

davispuh commented Aug 29, 2020

cvium commented Aug 30, 2020

		public bool EnableDecodingColorDepth10Hevc { get; set; }
		public bool EnableDecodingColorDepth10Vp9 { get; set; }

Add more separate hardware decoding toggles, support videotoolbox #2809

Add more separate hardware decoding toggles, support videotoolbox #2809

Conversation

nyanmisaka commented Apr 8, 2020

Artiume commented Apr 15, 2020

nyanmisaka commented Apr 15, 2020

JustAMan commented Apr 20, 2020

JustAMan left a comment

Choose a reason for hiding this comment

JustAMan commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

nyanmisaka commented Apr 20, 2020

JustAMan commented Apr 20, 2020

EraYaN commented May 13, 2020 • edited Loading

Artiume commented May 25, 2020 • edited Loading

rigtorp commented May 29, 2020

rigtorp commented May 29, 2020

Artiume commented May 29, 2020

Artiume commented May 29, 2020

crobibero Jun 2, 2020

Choose a reason for hiding this comment

Artiume commented Jun 2, 2020

JustAMan Jun 8, 2020

Choose a reason for hiding this comment

Artiume commented Jun 10, 2020

azure-pipelines bot commented Jun 10, 2020

Bond-009 commented Jun 10, 2020

azure-pipelines bot commented Jun 10, 2020

davispuh commented Aug 29, 2020

cvium commented Aug 30, 2020

EraYaN commented May 13, 2020 •

edited

Loading

Artiume commented May 25, 2020 •

edited

Loading