Herbstluftwm still exhibits rounded corners + shadow bug.

[bndlfm]

Platform

Endeavour OS

GPU, drivers, and screen setup

Nvidia GTX 1660, nvidia-dkms 515.48.07-2 from Arch's extra repo.

From right to left: 21.5" 1080x1920@60hz (Rotate 90°; ForceCompositionPipeline=On), center is a 32" 2560x1440@144hz ("Gsync Compatible Mode enabled"; set as primary display in X11), finally another 24" 1080x1920@60hz (Rotate 90°; ForceCompositionPipeline=On).

Environment

WM = Herbstluftwm, DM = ly, DE is a splash of plasma with whatever I've found over the years to my preferences. Login Shell is bash, interactive shell used is Fish w/ Kitty terminal, Kitty has an option to set which shell it uses in it's config.

picom version

picom-jonaburg-fix vgit-f70d0,
I've also tried your other fork: picom-arian8j2-git and it behaves similarly but I don't know if the shadow rounded corner patch is merged into that fork.

Configuration:

#{{{ Backends Settings
experimental-backends = true;
backend = "glx";

# Try to detect WM windows (a non-override-redirect window with no 
# child that has 'WM_STATE') and mark them as active.
#
# mark-wmwin-focused = false
mark-wmwin-focused = true;

# Mark override-redirect windows that doesn't have a child window with 'WM_STATE' focused.
# mark-ovredir-focused = false
mark-ovredir-focused = true;

# Try to detect windows with rounded corners and don't consider them 
# shaped windows. The accuracy is not very high, unfortunately.

# detect-rounded-corners = false
detect-rounded-corners = true;

# Detect '_NET_WM_OPACITY' on client windows, useful for window managers
# not passing '_NET_WM_OPACITY' of client windows to frame windows.
#
# detect-client-opacity = false
detect-client-opacity = true;

# Specify refresh rate of the screen. If not specified or 0, picom will 
# try detecting this with X RandR extension.
#
#refresh-rate = 60
refresh-rate = 0

# Use EWMH '_NET_ACTIVE_WINDOW' to determine currently focused window, 
# rather than listening to 'FocusIn'/'FocusOut' event. Might have more accuracy, 
# provided that the WM supports it.
# use-ewmh-active-win = false

# Unredirect all windows if a full-screen opaque window is detected, 
# to maximize performance for full-screen windows. Known to cause flickering 
# when redirecting/unredirecting windows. paint-on-overlay may make the flickering less obvious.
unredir-if-possible = true

# Delay before unredirecting the window, in milliseconds. Defaults to 0.
unredir-if-possible-delay = 0

# Conditions of windows that shouldn't be considered full-screen for unredirecting screen.
# unredir-if-possible-exclude = []

# Use 'WM_TRANSIENT_FOR' to group windows, and consider windows 
# in the same group focused at the same time.
#
# detect-transient = false
detect-transient = true

# Use 'WM_CLIENT_LEADER' to group windows, and consider windows in the same 
# group focused at the same time. 'WM_TRANSIENT_FOR' has higher priority if 
# detect-transient is enabled, too.
#
# detect-client-leader = false
detect-client-leader = true

# Resize damaged region by a specific number of pixels. 
# A positive value enlarges it while a negative one shrinks it. 
# If the value is positive, those additional pixels will not be actually painted 
# to screen, only used in blur calculation, and such. (Due to technical limitations, 
# with use-damage, those pixels will still be incorrectly painted to screen.) 
# Primarily used to fix the line corruption issues of blur, 
# in which case you should use the blur radius value here 
# (e.g. with a 3x3 kernel, you should use `--resize-damage 1`, 
# with a 5x5 one you use `--resize-damage 2`, and so on). 
# May or may not work with *--glx-no-stencil*. Shrinking doesn't function correctly.
#
# resize-damage = 1

# Specify a list of conditions of windows that should be painted with inverted color. 
# Resource-hogging, and is not well tested.
#
# invert-color-include = []

# GLX backend: Avoid using stencil buffer, useful if you don't have a stencil buffer. 
# Might cause incorrect opacity when rendering transparent content (but never 
# practically happened) and may not work with blur-background. 
# My tests show a 15% performance boost. Recommended.
#
glx-no-stencil = true

# GLX backend: Avoid rebinding pixmap on window damage. 
# Probably could improve performance on rapid window content changes, 
# but is known to break things on some drivers (LLVMpipe, xf86-video-intel, etc.).
# Recommended if it works.
#
glx-no-rebind-pixmap = true

# Disable the use of damage information. 
# This cause the whole screen to be redrawn everytime, instead of the part of the screen
# has actually changed. Potentially degrades the performance, but might fix some artifacts.
# The opposing option is use-damage
#
use-damage = true

# Use X Sync fence to sync clients' draw calls, to make sure all draw 
# calls are finished before picom starts drawing. Needed on nvidia-drivers 
# with GLX backend for some users.

xrender-sync-fence = false

# GLX backend: Use specified GLSL fragment shader for rendering window contents. 
# See `compton-default-fshader-win.glsl` and `compton-fake-transparency-fshader-win.glsl` 
# in the source tree for examples.
#
# glx-fshader-win = ''

# Force all windows to be painted with blending. Useful if you 
# have a glx-fshader-win that could turn opaque pixels transparent.
#
# force-win-blend = false

# Do not use EWMH to detect fullscreen windows. 
# Reverts to checking if a window is fullscreen based only on its size and coordinates.

 no-ewmh-fullscreen = false

# Dimming bright windows so their brightness doesn't exceed this set value. 
# Brightness of a window is estimated by averaging all pixels in the window, 
# so this could comes with a performance hit. 
# Setting this to 1.0 disables this behaviour. Requires --use-damage to be disabled. (default: 1.0)
#
# max-brightness = 1.0

# Make transparent windows clip other windows like non-transparent windows do,
# instead of blending on top of them.
#
# transparent-clipping = false

# Set the log level. Possible values are:
#  "trace", "debug", "info", "warn", "error"
# in increasing level of importance. Case doesn't matter. 
# If using the "TRACE" log level, it's better to log into a file 
# using *--log-file*, since it can generate a huge stream of logs.
#
# log-level = "debug"
log-level = "info";

# Set the log file.
# If *--log-file* is never specified, logs will be written to stderr. 
# Otherwise, logs will to written to the given file, though some of the early 
# logs might still be written to the stderr. 
# When setting this option from the config file, it is recommended to use an absolute path.
#
# log-file = '/home/neko/picom.log'

# Show all X errors (for debugging)
 show-all-xerrors = true

# Write process ID to a file.
# write-pid-path = '/path/to/your/log/file'
#}}}
#{{{ Animations
##################################
#          Animations           #
#################################
# requires https://github.com/jonaburg/picom
# (These are also the default values)
transition-length = 300
transition-pow-x = 0.1
transition-pow-y = 0.1
transition-pow-w = 0.1
transition-pow-h = 0.1
size-transition = true
#}}}
#{{{ Wintypes
# Window type settings
# 
# 'WINDOW_TYPE' is one of the 15 window types defined in EWMH standard: 
#     "unknown", "desktop", "dock", "toolbar", "menu", "utility", 
#     "splash", "dialog", "normal", "dropdown_menu", "popup_menu", 
#     "tooltip", "notification", "combo", and "dnd".
# 
# Following per window-type options are available: ::
# 
#   fade, shadow:::
#     Controls window-type-specific shadow and fade settings.
# 
#   opacity:::
#     Controls default opacity of the window type.
# 
#   focus:::
#     Controls whether the window of this type is to be always considered focused. 
#     (By default, all window types except "normal" and "dialog" has this on.)
# 
#   full-shadow:::
#     Controls whether shadow is drawn under the parts of the window that you 
#     normally won't be able to see. Useful when the window has parts of it 
#     transparent, and you want shadows in those areas.
# 
#   redir-ignore:::
#     Controls whether this type of windows should cause screen to become 
#     redirected again after been unredirected. If you have unredir-if-possible
#     set, and doesn't want certain window to cause unnecessary screen redirection, 
#     you can set this to `true`.
#
wintypes:
{
  normal = { shadow = true; }
  tooltip = { fade = false; shadow = true; opacity = 0.75; focus = true; full-shadow = false; };
  dock = { fade = false; shadow = true; }
  dnd = { shadow = false; }
  popup_menu = { opacity = 0.8; }
  dropdown_menu = { opacity = 0.8; }
};
#}}}
#{{{ Rounded Corners
#################################
#        Rounded Corners        #
#################################
# requires: https://github.com/sdhand/compton or https://github.com/jonaburg/picom

corner-radius = 14.0;
rounded-corners-exclude = [
  #"window_type = 'normal'",
#  "class_g = 'awesome'",
#  "class_g = 'dmenu'",
#  "class_g = 'dwm'",
#  "class_g = 'URxvt'",
#  "class_g = 'XTerm'",
#  "class_g = 'kitty'",
#  "class_g = 'Alacritty'",
  "class_g = 'Polybar'",
#  "class_g = 'code-oss'",
#  "class_g = 'xmobar'",
#  "class_g = 'firefox'",
#  "class_g = 'Thunderbird'"
];
#round-borders = 1;
round-borders-exclude = [
  "class_g = 'xmobar'",
  "class_g = 'polybar'",
];

#}}}
#{{{ Shadows
#################################
#             Shadows           #
#################################


# Enabled client-side shadows on windows. Note desktop windows 
# (windows with '_NET_WM_WINDOW_TYPE_DESKTOP') never get shadow, 
# unless explicitly requested using the wintypes option.
#
# shadow = false
shadow = true;

# The blur radius for shadows, in pixels. (defaults to 12)
# shadow-radius = 12
shadow-radius = 7;

# The opacity of shadows. (0.0 - 1.0, defaults to 0.75)
# shadow-opacity = .75

# The left offset for shadows, in pixels. (defaults to -15)
# shadow-offset-x = -15
shadow-offset-x = -7;

# The top offset for shadows, in pixels. (defaults to -15)
# shadow-offset-y = -15
shadow-offset-y = -7;

# Avoid drawing shadows on dock/panel windows. This option is deprecated,
# you should use the *wintypes* option in your config file instead.
#
# no-dock-shadow = false

# Don't draw shadows on drag-and-drop windows. This option is deprecated, 
# you should use the *wintypes* option in your config file instead.
#
# no-dnd-shadow = false

# Red color value of shadow (0.0 - 1.0, defaults to 0).
# shadow-red = 0

# Green color value of shadow (0.0 - 1.0, defaults to 0).
# shadow-green = 0

# Blue color value of shadow (0.0 - 1.0, defaults to 0).
# shadow-blue = 0

# Do not paint shadows on shaped windows. Note shaped windows 
# here means windows setting its shape through X Shape extension. 
# Those using ARGB background is beyond our control. 
# Deprecated, use 
#   shadow-exclude = 'bounding_shaped'
# or 
#   shadow-exclude = 'bounding_shaped && !rounded_corners'
# instead.
#
# shadow-ignore-shaped = ''

# Specify a list of conditions of windows that should have no shadow.
#
# examples:
#   shadow-exclude = "n:e:Notification";
#
# shadow-exclude = []
shadow-exclude = [
  "name = 'Notification'",
  "class_g = 'Conky'",
  "class_g ?= 'Notify-osd'",
  "class_g = 'Cairo-clock'",
  "class_g = 'slop'",
  "class_g = 'xmobar'",
  "class_g = 'Polybar'",
  "_GTK_FRAME_EXTENTS@:c"
];
#}}}
#{{{ Fading 
#################################
#           Fading              #
#################################

# Fade windows in/out when opening/closing and when opacity changes,
# unless no-fading-openclose is used.
fading = false;
# fading = true;

# Opacity change between steps while fading in. (0.01 - 1.0, defaults to 0.028)
#fade-in-step = 0.028
fade-in-step = 0.0;

# Opacity change between steps while fading out. (0.01 - 1.0, defaults to 0.03)
#fade-out-step = 0.03
fade-out-step = 0.0;

# The time between steps in fade step, in milliseconds. (> 0, defaults to 10)
# fade-delta = 10

# Specify a list of conditions of windows that should not be faded.
# don't need this, we disable fading for all normal windows with wintypes: {}
fade-exclude = [
  "class_g = 'slop'"   # maim
]

# Do not fade on window open/close.
# no-fading-openclose = false

# Do not fade destroyed ARGB windows with WM frame. Workaround of bugs in Openbox, Fluxbox, etc.
# no-fading-destroyed-argb = false

#}}}
# Transparency {{{
#################################
#   Transparency / Opacity      #
#################################


# Opacity of inactive windows. (0.1 - 1.0, defaults to 1.0)
# inactive-opacity = 1
inactive-opacity = 1;

# Opacity of window titlebars and borders. (0.1 - 1.0, disabled by default)
# frame-opacity = 1.0
frame-opacity = 0.85;

# Default opacity for dropdown menus and popup menus. (0.0 - 1.0, defaults to 1.0)
# menu-opacity = 1.0

# Let inactive opacity set by -i override the '_NET_WM_OPACITY' values of windows.
# inactive-opacity-override = true
inactive-opacity-override = false;

# Default opacity for active windows. (0.0 - 1.0, defaults to 1.0)
active-opacity = 1.0;

# Dim inactive windows. (0.0 - 1.0, defaults to 0.0)
# inactive-dim = 0.0

# Specify a list of conditions of windows that should always be considered focused.
# focus-exclude = []
focus-exclude = [
  "class_g = 'Cairo-clock'",
  "class_g = 'Bar'",                    # lemonbar
  "class_g = 'slop'",                   # maim
  "class_g = 'Steam'"                   # Steam (dropdown menus cause application to fade)
];

# Use fixed inactive dim value, instead of adjusting according to window opacity.
# inactive-dim-fixed = 1.0

# Specify a list of opacity rules, in the format `PERCENT:PATTERN`, 
# like `50:name *= "Firefox"`. picom-trans is recommended over this. 
# Note we don't make any guarantee about possible conflicts with other 
# programs that set '_NET_WM_WINDOW_OPACITY' on frame or client windows.
# example:
#    opacity-rule = [ "80:class_g = 'URxvt'" ];
#
# opacity-rule = []
opacity-rule = [
  "00:class_g    = '_HERBST_FRAME'",
  "85:class_g    = 'Rofi'",
  "100:class_g   = 'kitty'",
  "90:class_g    = 'Polybar'",
  "100:class_g   = 'firefox'",
  "100:class_g  *= 'FreeTube'",
  "100:class_g  *= 'discord'",
	"100:class_g  ?= 'dolphin'"
];

# }}}
#{{{ Blur 
blur:
{
  method = "dual_kawase"; 
  strength = 5;
}

blur-background-exclude = [ "window_type = 'dock'", "window_type = 'desktop'", "class_g? = '_HERBST_FRAME'", "class_g ?= 'firefox'" ];
#}}}

Steps of reproduction

1. Install herbstluftwm, picom, enable rounded corners + default shadow settings(I believe, or they were default when I first installed picom)
2. See outline of a square frame with rounded corners enabled.

log-file

[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_SGI_video_sync.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_SGI_swap_control.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Missing GLX extension GLX_OML_sync_control.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Missing GLX extension GLX_MESA_swap_control.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_EXT_swap_control.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_EXT_texture_from_pixmap.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_ARB_create_context.
[ 06/29/2022 14:45:34.389 glx_has_extension INFO ] Found GLX extension GLX_EXT_buffer_age.
[ 06/29/2022 14:45:34.412 gl_has_extension INFO ] Missing GL extension GL_GREMEDY_string_marker.
[ 06/29/2022 14:45:34.412 gl_init INFO ] GL vendor is NVIDIA, don't use glFinish
[ 06/29/2022 14:45:34.413 generate_dual_kawase_params INFO ] blur-strength: 5 [.iter = 2, .offset = 4.250000]
[ 06/29/2022 14:45:34.413 glGetUniformLocationChecked INFO ] Failed to get location of uniform 'opacity'. This is normal when using custom shaders.
[ 06/29/2022 14:45:34.439 x_fence_sync ERROR ] Failed to trigger the fence (X error 136 XSyncBadFence request 134 minor 15 serial 1318)
[ 06/29/2022 14:45:34.439 paint_all_new ERROR ] x_fence_sync failed, xrender-sync-fence will be disabled from now on.

Other details

I suspect this has to do with how herbstluft handles tiling with windows and frames. It resembles i3 quite a bit but without horizontal/vertical tabs in it's frames. You can cycle multiple windows so that they are split vertically, horizontally, only one window in the frame stack is visible, or you can split them into what would be considered the traditional type of tiling, separate frames.

The picom-jonaburg-fix version acts even more bizare, where it sometimes appears to be casting shadows outwards from the rounded corners andat other times (after killing picom) simply clips the window border like picom's main fork is currently doing. I have tried setting _HERBST_FRAME to zero opacity/shadow-exclude but the bug still appears.

[Arian8j2]

Thanks for reporting issues, First of all, i just applied another man patch to jonaburg fork so i really doesn't know that much about picom, You should report this in official picom (they are already fixing round corner so it's better to just wait for them). But it's weird that even with wm round corner you got this again, I tried dwm round corner patch in the past and it hid picom broken round corner, But i think Herbstluftwm is taking another approach to round corners. Sorry i can't help 😞