[Steam] Signal changes from HDR to SDR when launching games

[flibitijibibo]

Can also confirm what others are seeing: In Steam the display recognizes an HDR signal, but a few seconds after the game window starts presenting it falls back to an SDR signal, happens with both SDR and HDR content. The swapchain image formats seem correct so I guess it's between the compositor and display engine, rather than any particular application and the compositor. Could composition in gamescope be causing this?

Originally posted by @flibitijibibo in #1149 (comment)

[flibitijibibo]

According to Josh this may be the Steam client disabling HDR on launch based on its preferences; however HDR preferences aren't exposed in gamepadui for anything other than Steam Deck, so this may technically be a steam-for-linux issue as well.

[flibitijibibo]

It looks like it's not Steam after all - modified libliftoff to print errors on test commits:

[Gamescope WSI] Creating swapchain for xid: 0x2c00002 - minImageCount: 3 - format: VK_FORMAT_A2B10G10R10_UNORM_PACK32 - colorspace: VK_COLOR_SPACE_HDR10_ST2084_EXT - flip: true
[Gamescope WSI] Created swapchain for xid: 0x2c00002 - imageCount: 3
sh: 1: xdg-screensaver: not found
drmModeAtomicCommit: Invalid argument
Computing plane allocation on output 0x22604a0
Stopped reusing previous plane allocation on output 0x22604a0 (had reused it 485 times)
Layers on CRTC 93 (6 total):
  Layer 0x2260c90:
    FB_ID = 152
    zpos = 0
    alpha = 65535
    SRC_X = 0.000000
    SRC_Y = 0.000000
    SRC_W = 3840.000000
    SRC_H = 2160.000000
    rotation = 1
    CRTC_X = +0
    CRTC_Y = +0
    CRTC_W = 3840
    CRTC_H = 2160
    AMD_PLANE_DEGAMMA_TF = 0
    AMD_PLANE_SHAPER_LUT = 0
    AMD_PLANE_SHAPER_TF = 0
    AMD_PLANE_LUT3D = 0
    AMD_PLANE_CTM = 0
    AMD_PLANE_BLEND_TF = 0
Disabling plane 72
Disabling plane 66
Disabling plane 60
Disabling plane 54
Disabling plane 102
Disabling plane 98
Disabling plane 94
Disabling plane 90
Disabling plane 84
Disabling plane 78
Performing allocation for plane 72 (1/10)
drmModeAtomicCommit: Invalid argument
 Layer 0x2260c90 -> plane 72: test-only commit failed (Invalid argument)
Performing allocation for plane 90 (8/10)
Performing allocation for plane 84 (9/10)
drmModeAtomicCommit: Invalid argument
 Layer 0x2260c90 -> plane 84: test-only commit failed (Invalid argument)
Performing allocation for plane 78 (10/10)
drmModeAtomicCommit: Invalid argument
 Layer 0x2260c90 -> plane 78: test-only commit failed (Invalid argument)
Found a better allocation with score=0
Found plane allocation for output 0x22604a0 (score: 0, candidate planes: 10, tests: 3):
xwm: Failed to prepare 1-layer flip (Invalid argument), trying again with previous mode if modeset needed
Cannot re-use previous allocation: FB info changed
Computing plane allocation on output 0x22604a0
Layers on CRTC 93 (6 total):
  Layer 0x2260c90:
    FB_ID = 147
    zpos = 0
    alpha = 65535
    SRC_X = 0.000000
    SRC_Y = 0.000000
    SRC_W = 3840.000000
    SRC_H = 2160.000000
    rotation = 1
    CRTC_X = +0
    CRTC_Y = +0
    CRTC_W = 3840
    CRTC_H = 2160
    AMD_PLANE_DEGAMMA_TF = 3
    AMD_PLANE_SHAPER_LUT = 162
    AMD_PLANE_SHAPER_TF = 10
    AMD_PLANE_LUT3D = 163
    AMD_PLANE_CTM = 0
    AMD_PLANE_BLEND_TF = 3
  Layer 0x2260fe0:
    FB_ID = 159
    AMD_PLANE_DEGAMMA_TF = 1
    AMD_PLANE_SHAPER_LUT = 160
    AMD_PLANE_SHAPER_TF = 8
    AMD_PLANE_LUT3D = 161
    AMD_PLANE_BLEND_TF = 3
    AMD_PLANE_CTM = 0
    zpos = 3
    alpha = 65535
    SRC_X = 0.000000
    SRC_Y = 0.000000
    SRC_W = 1920.000000
    SRC_H = 1080.000000
    rotation = 1
    CRTC_X = +0
    CRTC_Y = +0
    CRTC_W = 3840
    CRTC_H = 2160
Disabling plane 72
Disabling plane 66
Disabling plane 60
Disabling plane 54
Disabling plane 102
Disabling plane 98
Disabling plane 94
Disabling plane 90
Disabling plane 84
Disabling plane 78
Performing allocation for plane 72 (1/10)
drmModeAtomicCommit: Invalid argument
 Layer 0x2260c90 -> plane 72: test-only commit failed (Invalid argument)
drmModeAtomicCommit: Invalid argument
 Layer 0x2260fe0 -> plane 72: test-only commit failed (Invalid argument)
Performing allocation for plane 90 (8/10)
 Layer 0x2260c90 -> plane 90: has composited layer on top
Performing allocation for plane 84 (9/10)
 Layer 0x2260c90 -> plane 84: has composited layer on top
drmModeAtomicCommit: Invalid argument
 Layer 0x2260fe0 -> plane 84: test-only commit failed (Invalid argument)
Performing allocation for plane 78 (10/10)
 Layer 0x2260c90 -> plane 78: has composited layer on top
drmModeAtomicCommit: Invalid argument
 Layer 0x2260fe0 -> plane 78: test-only commit failed (Invalid argument)
Found a better allocation with score=0
Found plane allocation for output 0x22604a0 (score: 0, candidate planes: 10, tests: 4):
xwm: Failed to prepare 1-layer flip entirely: Invalid argument
[Gamescope WSI] Swapchain recieved new refresh cycle: 8.33ms
drm: Connector DP-1 -> SAM - QCQ90S
drm: [colorimetry]: EDID with colorimetry detected. Using it
drm: [colorimetry]: r 0.669922 0.311523
drm: [colorimetry]: g 0.273438 0.659180
drm: [colorimetry]: b 0.153320 0.044922
drm: [colorimetry]: w 0.312500 0.329102
drm: selecting connector DP-1
drm: selecting mode 3840x2160@120Hz
Cannot re-use previous allocation: a layer has been added or removed
Computing plane allocation on output 0x7f61902f1840
Layers on CRTC 93 (6 total):
  Layer 0x7f61902f23e0:
    FB_ID = 153
    zpos = 0
    alpha = 65535
    SRC_X = 0.000000
    SRC_Y = 0.000000
    SRC_W = 3840.000000
    SRC_H = 2160.000000
    rotation = 1
    CRTC_X = +0
    CRTC_Y = +0
    CRTC_W = 3840
    CRTC_H = 2160
    AMD_PLANE_DEGAMMA_TF = 0
    AMD_PLANE_SHAPER_LUT = 0
    AMD_PLANE_SHAPER_TF = 0
    AMD_PLANE_LUT3D = 0
    AMD_PLANE_CTM = 0
    AMD_PLANE_BLEND_TF = 0
Disabling plane 72
Disabling plane 66
Disabling plane 60
Disabling plane 54
Disabling plane 102
Disabling plane 98
Disabling plane 94
Disabling plane 90
Disabling plane 84
Disabling plane 78
Performing allocation for plane 72 (1/10)
drmModeAtomicCommit: Success
 Layer 0x7f61902f23e0 -> plane 72: success
  Performing allocation for plane 90 (8/10)
  Performing allocation for plane 84 (9/10)
  Performing allocation for plane 78 (10/10)
  Found a better allocation with score=1
Performing allocation for plane 90 (8/10)
Performing allocation for plane 84 (9/10)
drmModeAtomicCommit: Invalid argument
 Layer 0x7f61902f23e0 -> plane 84: test-only commit failed (Invalid argument)
Found plane allocation for output 0x7f61902f1840 (score: 1, candidate planes: 10, tests: 2):
  Layer 0x7f61902f23e0 -> plane 72
drmModeAtomicCommit: Success
Reusing previous plane allocation on output 0x7f61902f1840
drmModeAtomicCommit: Success

So it looks like we're down to drm; it's trying to reallocate the planes for the game and it fails, and eventually falls back to SDR.

[MarijnS95]

This is likely fixed by 0502c67. I faced the exact same HDR->SDR problem a few weeks ago while HDR in games was working fine a few months ago, and ran into this conflict while rebasing my local hack/workaround to fix this "regression" introduced by #1085 (sorry for not reporting this in time!).

Unfortunately that commit barely missed out on the 3.14.3 tag!

(For those interested: if a game via swapchain feedback provides no or invalid HDR metadata, the code after #1085 wouldn't configure the HDR_OUTPUT_METADATA DRM property which effectively disables HDR output. Before, and after 0502c67, that code would fall back to default metadata generated based on the display color space and properties)

[flibitijibibo]

Can confirm that 0502c67 resolves the issue.