KMS: incorrect gamma / overly bright image with 2 monitors

[HiassofT]

Describe the bug
When running kernel 5.10.76 on RPi4 with KMS and two HDMI monitors connected the gamma values seem to be completely off and the displayed image is overly bright and has wrong colors.

This doesn't happen with just a single HDMI monitor connected or when using kernel 5.10.75 or earlier.

To reproduce

• boot RPiOS lite on RPi4 with 2 HDMI monitors connected
• run modetest -s ... eg modetest -s 32:1920x1200-60

Expected behaviour
The test pattern is shown with correct colors:

Actual behaviour
The test pattern is shown with wrong colors:

System
RPi4, initially reported and reproduced on LibreELEC but the same happens on RPi-OS, too. eg with current rpi-update kernel:

Linux raspberrypi 5.10.76-v7l+ #1477 SMP Tue Nov 2 13:20:53 GMT 2021 armv7l GNU/Linux
Nov  2 2021 13:22:15 
Copyright (c) 2012 Broadcom
version e50fe24ee2b6974f3ba6615ba0f1d8f45c485f69 (clean) (release) (start)

rpi-update kernel 992de83e0ea01a9765196349abb386c5c8ea8944 has this issue, too

Linux raspberrypi 5.10.76-v7l+ #1474 SMP Thu Oct 28 13:42:00 BST 2021 armv7l GNU/Linux
Oct 28 2021 14:14:14 
Copyright (c) 2012 Broadcom
version bcdfe55d56b75b4be52c10d73bdb4a60e2b49f75 (clean) (release) (start)

but c59a637736f579733dfed9c62235c180471bbcba is OK

Linux raspberrypi 5.10.75-v7l+ #1473 SMP Mon Oct 25 15:18:06 BST 2021 armv7l GNU/Linux
Oct 25 2021 14:59:11
Copyright (c) 2012 Broadcom
version f1b25b0ca39fec42e8590287ad12c23f821ad7fd (clean) (release) (start)

Additional context
With a single monitor connected hvs_gamma in debugfs looks OK:

HVS channel 0:
  red:
  00000000 00000100 - x 0, c 0, grad 256
  10001000 00000100 - x 256, c 4096, grad 256
  20002000 00000100 - x 512, c 8192, grad 256
  30003000 00000100 - x 768, c 12288, grad 256
  40004000 00000100 - x 1024, c 16384, grad 256
  50005000 00000100 - x 1280, c 20480, grad 256
  60006000 00000100 - x 1536, c 24576, grad 256
  70007000 00000100 - x 1792, c 28672, grad 256
  80008000 00000100 - x 2048, c 32768, grad 256
  90009000 00000100 - x 2304, c 36864, grad 256
  a000a000 00000100 - x 2560, c 40960, grad 256
  b000b000 00000100 - x 2816, c 45056, grad 256
  c000c000 00000100 - x 3072, c 49152, grad 256
  d000d000 00000100 - x 3328, c 53248, grad 256
  e000e000 00000100 - x 3584, c 57344, grad 256
  f000f000 00000100 - x 3840, c 61440, grad 256
  green:
  00000000 00000100 - x 0, c 0, grad 256
  10001000 00000100 - x 256, c 4096, grad 256
  20002000 00000100 - x 512, c 8192, grad 256
  30003000 00000100 - x 768, c 12288, grad 256
  40004000 00000100 - x 1024, c 16384, grad 256
  50005000 00000100 - x 1280, c 20480, grad 256
  60006000 00000100 - x 1536, c 24576, grad 256
  70007000 00000100 - x 1792, c 28672, grad 256
  80008000 00000100 - x 2048, c 32768, grad 256
  90009000 00000100 - x 2304, c 36864, grad 256
  a000a000 00000100 - x 2560, c 40960, grad 256
  b000b000 00000100 - x 2816, c 45056, grad 256
  c000c000 00000100 - x 3072, c 49152, grad 256
  d000d000 00000100 - x 3328, c 53248, grad 256
  e000e000 00000100 - x 3584, c 57344, grad 256
  f000f000 00000100 - x 3840, c 61440, grad 256
  blue:
  00000000 00000100 - x 0, c 0, grad 256
  10001000 00000100 - x 256, c 4096, grad 256
  20002000 00000100 - x 512, c 8192, grad 256
  30003000 00000100 - x 768, c 12288, grad 256
  40004000 00000100 - x 1024, c 16384, grad 256
  50005000 00000100 - x 1280, c 20480, grad 256
  60006000 00000100 - x 1536, c 24576, grad 256
  70007000 00000100 - x 1792, c 28672, grad 256
  80008000 00000100 - x 2048, c 32768, grad 256
  90009000 00000100 - x 2304, c 36864, grad 256
  a000a000 00000100 - x 2560, c 40960, grad 256
  b000b000 00000100 - x 2816, c 45056, grad 256
  c000c000 00000100 - x 3072, c 49152, grad 256
  d000d000 00000100 - x 3328, c 53248, grad 256
  e000e000 00000100 - x 3584, c 57344, grad 256
  f000f000 00000100 - x 3840, c 61440, grad 256
HVS channel 1: Channel disabled
HVS channel 2: Channel disabled

with two monitors the values in hvs_gamma look very odd

HVS channel 0:
  red:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  green:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  blue:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
HVS channel 1:
  red:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  green:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  blue:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
HVS channel 2: Channel disabled

ping @6by9

[popcornmix]

With 2 monitors I get this warn when booting to desktop following the gamma PRs:

[   20.867733] ------------[ cut here ]------------
[   20.867756] WARNING: CPU: 3 PID: 577 at lib/refcount.c:28 refcount_warn_saturate+0x13c/0x174
[   20.867762] refcount_t: underflow; use-after-free.
[   20.867768] Modules linked in: 8021q garp stp llc snd_soc_hdmi_codec joydev brcmfmac brcmutil v3d gpu_sched cfg80211 raspberrypi_hwmon vc4 rfkill i2c_brcmstb cec drm_kms_helper snd_soc_core bcm2835_v4l2(C) snd_bcm2835(C) videobuf2_vmalloc snd_compress rpivid_hevc(C) bcm2835_codec(C) snd_pcm_dmaengine bcm2835_isp(C) snd_pcm v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_memops snd_timer videobuf2_v4l2 videobuf2_common vc_sm_cma(C) videodev snd syscopyarea sysfillrect mc sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio i2c_dev fuse drm drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[   20.868095] CPU: 3 PID: 577 Comm: Xorg Tainted: G         C        5.10.77-v7l+ #265
[   20.868099] Hardware name: BCM2711
[   20.868104] Backtrace: 
[   20.868118] [<c0b9b154>] (dump_backtrace) from [<c0b9b4e0>] (show_stack+0x20/0x24)
[   20.868124]  r7:ffffffff r6:00000000 r5:60000013 r4:c12e6f58
[   20.868132] [<c0b9b4c0>] (show_stack) from [<c0b9f9ac>] (dump_stack+0xc8/0xf4)
[   20.868140] [<c0b9f8e4>] (dump_stack) from [<c0221138>] (__warn+0xfc/0x114)
[   20.868147]  r10:c3c97e64 r9:c2b7b800 r8:00000009 r7:0000001c r6:c073d170 r5:00000009
[   20.868151]  r4:c0e6aef0 r3:c1205094
[   20.868158] [<c022103c>] (__warn) from [<c0b9bc84>] (warn_slowpath_fmt+0xa8/0xe8)
[   20.868164]  r7:c073d170 r6:0000001c r5:c0e6aef0 r4:c0e6af2c
[   20.868171] [<c0b9bbe0>] (warn_slowpath_fmt) from [<c073d170>] (refcount_warn_saturate+0x13c/0x174)
[   20.868177]  r8:c3c96000 r7:c3dd37c0 r6:c3cf2074 r5:c3f2e00c r4:c3f2e000
[   20.868283] [<c073d034>] (refcount_warn_saturate) from [<bf0d4648>] (drm_mode_object_put.part.0+0x70/0x84 [drm])
[   20.868445] [<bf0d45d8>] (drm_mode_object_put.part.0 [drm]) from [<bf0d4680>] (drm_mode_object_put+0x24/0x28 [drm])
[   20.868450]  r5:00000000 r4:00000000
[   20.868608] [<bf0d465c>] (drm_mode_object_put [drm]) from [<bf0d5484>] (drm_property_blob_put+0x20/0x24 [drm])
[   20.868765] [<bf0d5464>] (drm_property_blob_put [drm]) from [<bf0d8dec>] (drm_mode_gamma_set_ioctl+0x3c4/0x524 [drm])
[   20.868923] [<bf0d8a28>] (drm_mode_gamma_set_ioctl [drm]) from [<bf0ba5fc>] (drm_ioctl_kernel+0xc8/0x10c [drm])
[   20.868930]  r10:000000a5 r9:c325d400 r8:c3c97e64 r7:bf0d8a28 r6:00000002 r5:c2b7b800
[   20.868934]  r4:00000000
[   20.869091] [<bf0ba534>] (drm_ioctl_kernel [drm]) from [<bf0ba840>] (drm_ioctl+0x200/0x3d4 [drm])
[   20.869097]  r9:c325d400 r8:c3c97e64 r7:00000020 r6:bf0f0acc r5:00000020 r4:c02064a5
[   20.869185] [<bf0ba640>] (drm_ioctl [drm]) from [<c0457b48>] (sys_ioctl+0x120/0x934)
[   20.869191]  r10:c3347428 r9:0000000c r8:c2d66f00 r7:becdfa28 r6:c2d66f01 r5:00000000
[   20.869195]  r4:c02064a5
[   20.869204] [<c0457a28>] (sys_ioctl) from [<c0200040>] (ret_fast_syscall+0x0/0x28)
[   20.869209] Exception stack(0xc3c97fa8 to 0xc3c97ff0)
[   20.869214] 7fa0:                   00000000 becdfa28 0000000c c02064a5 becdfa28 019b23b8
[   20.869220] 7fc0: 00000000 becdfa28 c02064a5 00000036 00000000 019b1b84 019b25c0 becdfa94
[   20.869225] 7fe0: b6d8308c becdfa0c b6d68c5c b6a231ac
[   20.869231]  r10:00000036 r9:c3c96000 r8:c0200204 r7:00000036 r6:c02064a5 r5:becdfa28
[   20.869235]  r4:00000000
[   20.869251] ---[ end trace dbf934b87bdf1973 ]---

If I revert #4664 the WARN is gone and the brightness is fixed.

EDIT: brightness was wrong on a reboot. I do need to revert #4435 too for brightness fix.

[6by9]

Two independent things - please be careful not to conflate issues.

vc5 HVS keeps the gamma block enabled as it isn't safe to enable/disable it randomly (although as we're now always doing a mode switch it may be possible). It isn't clearing the gamma table though, so it is enabled with whatever was left in the SRAMs.

Refcounting I have no idea on at present.

[6by9]

Doh, refcounting is stupid.
There's no return after the drm_atomic_commit, so we do call drm_property_blob_put twice.

[HiassofT]

Another data point: Gamma is also off with a single monitor connected to HDMI1.

Tested with rpi-update kernel raspberrypi/rpi-firmware@992de83 (so just gamma LUT support, but not the recent change check commit) and modetest -s 40:1920x1200-60

Right after boot I get these hvs_gamma values:

root@raspberrypi:~# cat /sys/kernel/debug/dri/1/hvs_gamma 
HVS channel 0:
  red:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  green:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
  blue:
  00000000 00000c23 - x 0, c 0, grad 3107
  01500fee 000006d7 - x 21, c 4078, grad 1751
  03a01fbf 0000042f - x 58, c 8127, grad 1071
  07802ff6 000002e8 - x 120, c 12278, grad 744
  0d003ff0 0000022e - x 208, c 16368, grad 558
  14504fe2 000001b8 - x 325, c 20450, grad 440
  1db05ffb 00000166 - x 475, c 24571, grad 358
  29206ffd 0000012c - x 658, c 28669, grad 300
  36c07ff4 00000100 - x 876, c 32756, grad 256
  46c08ff7 000000de - x 1132, c 36855, grad 222
  59309ffa 000000c3 - x 1427, c 40954, grad 195
  6e20aff6 000000ae - x 1762, c 45046, grad 174
  85c0bffd 0000009c - x 2140, c 49149, grad 156
  a000cff7 0000008d - x 2560, c 53239, grad 141
  bd20dffd 00000080 - x 3026, c 57341, grad 128
  dd10effb 00000075 - x 3537, c 61435, grad 117
HVS channel 1: Channel disabled
HVS channel 2: Channel disabled

[6by9]

The PWLs are initialised in vc4_crtc_init https://github.com/raspberrypi/linux/blob/rpi-5.10.y/drivers/gpu/drm/vc4/vc4_crtc.c#L1238

The issue seems to be that the gamma block is not in a state to be written to from vc4_hvs_init_channel (called from vc4_hvs_atomic_enable).
It is happy when written from vc4_hvs_atomic_flush, but that only writes it if crtc->state->color_mgmt_changed is set, so in the case where we initialise with no gamma lut, it isn't written.

As we always disable/enable the pipeline for gamma changes on 2711, I'm tempted to disable the gamma block on demand as mentioned above.
I need to do a 2710 build as well to check whether we can dynamically enable/disable the gamma block there or not. The current code implies that we can (and do), but it's not totally clear.