Opportunity for a new default tonemap mode

[gnattu]

Jellyfin currently has two tonemap modes: MAX mode, which applies TMO to the maximum of the RGB channels, and RGB mode, which applies TMO to each RGB channel individually. However, both methods have significant shortcomings in certain cases. A new mode that applies TMO on the relative luminance has recently been merged in #418 and #416. This mode resolves the distorted color/overly saturated color issue in RGB mode and the significant hue shift issue on bright pixels in MAX mode.

My current working draft introduces a new mode called ITP mode. This mode does not operate in the linear RGB color space like the other methods but operates in the ICtCp color space. The TMO is applied to the Intensity of the ICtCp color. This mode also has a special desaturation method that preserves hue in ICtCp space, which should be closer to human perception due to the design of this color space. This tonemap mode preserves colors in bright pixels even better than the LUM mode, and I propose to make this mode the new default for Jellyfin 10.10 and jellyfin-ffmpeg 7.0.

The major drawback of this mode is the extra computation required for the color space conversion. However, for BT.2390 methods, this is not much slower because this color space does not need the extra non-linearity applied to the signal like other modes, as such non-linearity functions are already applied during color space conversion. For other TMOs that expect a linear signal, extra linearization and de-linearization are required, which makes this mode slower. For users with lower spec GPUs, I suggest them to use the LUM mode which is a good enough alternative for common cases.

To demonstrate the effectiveness of this tonemap mode, I made some sample scene images as examples.

Let's begin with the customizable desaturation parameter. Look at the following furnace example with desat=0.5:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

As we can see from this example, ICtCp-based desaturation shows a significant advantage over RGB-based desaturation. LUM and MAX modes have a significant hue shift, causing the furnace to appear too red. The RGB mode does not experience as severe a hue shift, but the desaturation is not effective as the color remains overly saturated. ITP mode preserves the hue while gradually turning the overly bright furnace color to white, producing most naturally looking color.

In fact, for this scene, the ICtCp-based tone mapping does not have over saturation problem in the first place, I made this sample mostly to show the hue preserving capability of ICtCp. This is the results of desat=0:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

As you can see, the ITP mode produces natural-looking colors and does not require extra desaturation.

---

To demonstrate the detail-preserving capability for bright pixels, let look at this glass blowing scene:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

As you can see, RGB mode preserves highlights but produces unnatural colors. MAX mode loses most of the highlights, and while LUM mode preserves highlights better than MAX, it is still not as effective as ITP mode.

Another example is the campfire scene:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

Pay attention to the details of the flame in the bottom right corner.

---

To further demonstrate the limitation of current modes (MAX and RGB), lets look at this scene that direct captures the sun:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

As you can see, both of the current modes render sunlight poorly. Both LUM and ITP modes improve the sunlight significantly. In this case, LUM mode has a darker sun border, which appears less "natural" but is not subjectively bad.

Direct capture of the sun is an extreme case, but the current modes have more limitations. Let's look at this bridge scene shot at night:

RGB mode:

MAX mode:

LUM mode:

ITP mode:

RGB mode is too dark for a night scene, and MAX mode renders the lights on the road way too red. Both LUM and ITP modes produce much more natural colors in this case. For this scene, the LUM and ITP results are not visually distinguishable.

For more scene comparisons, I have made a zip file that includes additional scenes not shown here:tonemode_comparison_extra.zip

If there is no objection I will make the ITP mode as the default tonemap mode for jellyfin-ffmpeg7 and Jellyfin 10.10.

[nyanmisaka]

No objection to enabling new method/mode by default if it provides a better look and feel, and falls back to using LUM instead on weak GPUs. But your RGB mode seems off to me.

[gnattu]

No objection to enabling new method/mode by default if it provides a better look and feel, and falls back to using LUM instead on weak GPUs. But your RGB mode seems off to me.

Yep, the original one divided too deep factor and yours are correct, and RGB mode is problematic when desired to retain bright saturated colors as the hue shift is also sever at those colors.:

RGB:

ITP:

[nyanmisaka]

BTW my suggestion in #416 may be wrong, because primaries have been converted from bt2020 to bt709 in lrgb2lrgb() when doing hdr2sdr. The luminance coeffs should be determined by the setting of the :p=bt2020/bt709 option if we want to do tone-mapping in the target primaries instead of the source primaries.

[gnattu]

because primaries have been converted from bt2020 to bt709 in lrgb2lrgb() when doing hdr2sdr

That's why I modified the shaders further. In my latest draft ITP and LUM mode will now map in source space and perform lrgb to lrgb after tone mapping(partially addressed nyanmisaka#5)

The current draft has Metal, OpenCL and CUDA modified for this change, but the actual PR is blocked by the ffmpeg7 PR as the patches are based on that.

[nyanmisaka]

because primaries have been converted from bt2020 to bt709 in lrgb2lrgb() when doing hdr2sdr

That's why I modified the shaders further. In my latest draft ITP and LUM mode will now map in source space and perform lrgb to lrgb after tone mapping(partially addressed nyanmisaka#5)

The current draft has Metal, OpenCL and CUDA modified for this change, but the actual PR is blocked by the ffmpeg7 PR as the patches are based on that.

The vulkan changes are more problematic than I expected. Let's merge this PR into the jellyfin-7.0 branch so you can continue working on it.