Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

32 bit ICD loading seems broken as of 570.86.16 #173

Closed
drelbszoomer opened this issue Feb 11, 2025 · 11 comments
Closed

32 bit ICD loading seems broken as of 570.86.16 #173

drelbszoomer opened this issue Feb 11, 2025 · 11 comments

Comments

@drelbszoomer
Copy link

drelbszoomer commented Feb 11, 2025 •
edited
Loading 

uname -a                                                                                                                                                                                                                               
Linux 6.12.12-203.bazzite.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb  4 23:54:25 UTC 2025 x86_64 GNU/Linux

    dnf5 install -y \
        libnvidia-fbc \
        libnvidia-ml.i686 \
        libva-nvidia-driver \
        nvidia-driver \
        nvidia-driver-cuda \
        nvidia-driver-cuda-libs.i686 \
        nvidia-driver-libs.i686 \
        vulkan-loader.i686 \
        nvidia-settings \

cat /proc/driver/nvidia/version                                                                                                                                                                                                        

NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.86.16  Fri Jan 24 21:25:51 UTC 2025
GCC version:  gcc version 14.2.1 20250110 (Red Hat 14.2.1-7) (GCC) 


ls /usr/share/vulkan/icd.d/nvidia_icd*                                                                                                                                                                                                 
/usr/share/vulkan/icd.d/nvidia_icd.i686.json  /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json

cat /usr/share/vulkan/icd.d/nvidia_icd*     
                                                                                                                                                                                          
{
    "file_format_version" : "1.0.1",
    "ICD": {
        "library_path": "/usr/lib/libGLX_nvidia.so.0",
        "api_version" : "1.4.303"
    }
}
{
    "file_format_version" : "1.0.1",
    "ICD": {
        "library_path": "/usr/lib64/libGLX_nvidia.so.0",
        "api_version" : "1.4.303"
    }
}

vulkaninfo --summary                                                                                                                                                                                                                   

WARNING: [Loader Message] Code 0 : Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.304


Instance Extensions: count = 24
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 2
--------------------------
VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1
VK_LAYER_NV_optimus         NVIDIA Optimus layer         1.4.303  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.303
	driverVersion      = 570.86.16.0
	vendorID           = 0x10de
	deviceID           = 0x1e89
	deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
	deviceName         = NVIDIA GeForce RTX 2060
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 570.86.16
	conformanceVersion = 1.4.1.0
	deviceUUID         = 925d432e-5317-d686-9fd3-93fe18cf833d
	driverUUID         = 5429646d-3d95-54e1-b807-75d58fc2ef85

# RPM source: https://kojihub.stream.centos.org/koji/buildinfo?buildID=46408
$ curl -LO https://kojihub.stream.centos.org/kojifiles/packages/vulkan-tools/1.3.268.0/1.el9/i686/vulkan-tools-1.3.268.0-1.el9.i686.rpm
$ bsdtar -xf vulkan-tools-1.3.268.0-1.el9.i686.rpm --strip-components 3 --include usr/bin/vulkaninfo -O > vulkaninfo32
$ chmod +x vulkaninfo32
$ rm vulkan-tools-1.3.268.0-1.el9.i686.rpm

 ./vulkaninfo32 --summary      
                                                                                                                                                                                                         
ERROR: [Loader Message] Code 0 : vkCreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /builddir/build/BUILD/Vulkan-Tools-vulkan-sdk-1.3.268.0/vulkaninfo/vulkaninfo.h:688:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

VK_LOADER_DEBUG=all shows that vulkaninfo32 never tries to load the i686 icd, it just tries to load the x86_64 one (which throws an ELF mismatch error as it's the wrong type)

I notice that the nvidia flatpak extensions recently stopped installing 2x icds here: flathub/org.freedesktop.Platform.GL.nvidia@d55b7cb (the above stuff was all done outside of a flatpak, but that's where I first noticed the failure and ultimately ended up reproducing the issue outside of a flatpak env).

I also notice that at one point, a PR was done to the .spec file to use a single ICD: #95

but this was reverted.

Did something change recently? Should one ICD be used here?
@drelbszoomer drelbszoomer changed the title 32 bit ICD loading seems broken 32 bit ICD loading seems broken as of 570.86.16 Feb 11, 2025
@scaronni
Copy link
Member

Thanks for reporting.

 ./vulkaninfo32 --summary      
                                                                                                                                                                                                         
ERROR: [Loader Message] Code 0 : vkCreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /builddir/build/BUILD/Vulkan-Tools-vulkan-sdk-1.3.268.0/vulkaninfo/vulkaninfo.h:688:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

VK_LOADER_DEBUG=all shows that vulkaninfo32 never tries to load the i686 icd, it just tries to load the x86_64 one (which throws an ELF mismatch error as it's the wrong type)

Doh, this is interesting, because it's what the other drivers are doing:

$ ls -al /usr/share/vulkan/icd.d/
total 88
drwxr-xr-x. 1 root root 984 Feb  9 10:07 .
drwxr-xr-x. 1 root root  74 Jan 15 01:00 ..
-rw-r--r--. 1 root root 149 Feb  8 01:00 broadcom_icd.i686.json
-rw-r--r--. 1 root root 151 Feb  8 01:00 broadcom_icd.x86_64.json
-rw-r--r--. 1 root root 150 Feb  8 01:00 freedreno_icd.i686.json
-rw-r--r--. 1 root root 152 Feb  8 01:00 freedreno_icd.x86_64.json
-rw-r--r--. 1 root root 152 Feb  8 01:00 intel_hasvk_icd.i686.json
-rw-r--r--. 1 root root 154 Feb  8 01:00 intel_hasvk_icd.x86_64.json
-rw-r--r--. 1 root root 146 Feb  8 01:00 intel_icd.i686.json
-rw-r--r--. 1 root root 148 Feb  8 01:00 intel_icd.x86_64.json
-rw-r--r--. 1 root root 144 Feb  8 01:00 lvp_icd.i686.json
-rw-r--r--. 1 root root 146 Feb  8 01:00 lvp_icd.x86_64.json
-rw-r--r--. 1 root root 148 Feb  8 01:00 nouveau_icd.i686.json
-rw-r--r--. 1 root root 150 Feb  8 01:00 nouveau_icd.x86_64.json
-rw-r--r--. 1 root root 149 Feb  6 01:00 nvidia_icd.i686.json
-rw-r--r--. 1 root root 151 Feb  6 01:00 nvidia_icd.x86_64.json
-rw-r--r--. 1 root root 149 Feb  8 01:00 panfrost_icd.i686.json
-rw-r--r--. 1 root root 151 Feb  8 01:00 panfrost_icd.x86_64.json
-rw-r--r--. 1 root root 153 Feb  8 01:00 powervr_mesa_icd.i686.json
-rw-r--r--. 1 root root 155 Feb  8 01:00 powervr_mesa_icd.x86_64.json
-rw-r--r--. 1 root root 147 Feb  8 01:00 radeon_icd.i686.json
-rw-r--r--. 1 root root 149 Feb  8 01:00 radeon_icd.x86_64.json
-rw-r--r--. 1 root root 147 Feb  8 01:00 virtio_icd.i686.json
-rw-r--r--. 1 root root 149 Feb  8 01:00 virtio_icd.x86_64.json

And they are following the same approach for both 32 and 64 bit by listing the library with the full path:

$ cat /usr/share/vulkan/icd.d/radeon_icd.x86_64.json 
{
    "ICD": {
        "api_version": "1.3.296",
        "library_path": "/usr/lib64/libvulkan_radeon.so"
    },
    "file_format_version": "1.0.0"
}
$ cat /usr/share/vulkan/icd.d/radeon_icd.i686.json 
{
    "ICD": {
        "api_version": "1.3.296",
        "library_path": "/usr/lib/libvulkan_radeon.so"
    },
    "file_format_version": "1.0.0"

I notice that the nvidia flatpak extensions recently stopped installing 2x icds here: flathub/org.freedesktop.Platform.GL.nvidia@d55b7cb (the above stuff was all done outside of a flatpak, but that's where I first noticed the failure and ultimately ended up reproducing the issue outside of a flatpak env).

I also notice that at one point, a PR was done to the .spec file to use a single ICD: #95

but this was reverted.

Did something change recently? Should one ICD be used here?

@scaronni
Copy link
Member

One single ICD (nvidia_icd.json) is fine because in there the library is not listed with the full path, so I actually add the full path while installing:

https://github.com/negativo17/nvidia-driver/blob/master/nvidia-driver.spec#L203-L205

@scaronni
Copy link
Member

In fact I don't have this issue on Fedora:

$ file vulkaninfo 
vulkaninfo: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=4bf417c977efce1d896f97da52bcc483ca41ee05, for GNU/Linux 3.2.0, stripped

$ ./vulkaninfo --summary
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
'DISPLAY' environment variable not set... skipping surface info
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.304


Instance Extensions: count = 25
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 12
---------------------------
VK_LAYER_FROG_gamescope_wsi_x86_64 Gamescope WSI (XWayland Bypass) Layer (x86_64) 1.3.221  version 1
VK_LAYER_MANGOAPP_overlay          Mangoapp Layer                                 1.3.0    version 1
VK_LAYER_MANGOAPP_overlay          Mangoapp Layer                                 1.3.0    version 1
VK_LAYER_MANGOHUD_overlay_x86      Vulkan Hud Overlay                             1.3.0    version 1
VK_LAYER_MANGOHUD_overlay_x86_64   Vulkan Hud Overlay                             1.3.0    version 1
VK_LAYER_MESA_device_select        Linux device selection layer                   1.3.211  version 1
VK_LAYER_NV_optimus                NVIDIA Optimus layer                           1.4.303  version 1
VK_LAYER_VALVE_steam_fossilize_32  Steam Pipeline Caching Layer                   1.3.207  version 1
VK_LAYER_VALVE_steam_fossilize_64  Steam Pipeline Caching Layer                   1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_32    Steam Overlay Layer                            1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_64    Steam Overlay Layer                            1.3.207  version 1
VK_LAYER_hdr_wsi                   WSI HDR Layer (x86_64)                         1.3.221  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.303
	driverVersion      = 570.116.0.0
	vendorID           = 0x10de
	deviceID           = 0x2783
	deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
	deviceName         = NVIDIA GeForce RTX 4070 SUPER
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 570.116
	conformanceVersion = 1.4.1.0
	deviceUUID         = bd66b31f-f46a-7852-3408-9e9249d64cbe
	driverUUID         = ba763729-60ed-544e-b92f-bcc1ed57d283
GPU1:
	apiVersion         = 1.3.296
	driverVersion      = 24.3.4
	vendorID           = 0x1002
	deviceID           = 0x164e
	deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
	deviceName         = AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO)
	driverID           = DRIVER_ID_MESA_RADV
	driverName         = radv
	driverInfo         = Mesa 24.3.4
	conformanceVersion = 1.3.0.0
	deviceUUID         = 00000000-6b00-0000-0000-000000000000
	driverUUID         = 414d442d-4d45-5341-2d44-525600000000
GPU2:
	apiVersion         = 1.3.296
	driverVersion      = 0.0.1
	vendorID           = 0x10005
	deviceID           = 0x0000
	deviceType         = PHYSICAL_DEVICE_TYPE_CPU
	deviceName         = llvmpipe (LLVM 19.1.7, 256 bits)
	driverID           = DRIVER_ID_MESA_LLVMPIPE
	driverName         = llvmpipe
	driverInfo         = Mesa 24.3.4 (LLVM 19.1.7)
	conformanceVersion = 1.3.1.1
	deviceUUID         = 6d657361-3234-2e33-2e34-000000000000
	driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Even with a quite old vulkaninfo command:

$ strings vulkaninfo | grep fc39
{"type":"rpm","name":"vulkan-tools","version":"1.3.268.0-1.fc39","architecture":"i386","osCpe":"cpe:/o:fedoraproject:fedora:39"}
vulkaninfo-1.3.268.0-1.fc39.i386.debug

@scaronni
Copy link
Member

scaronni commented Feb 11, 2025 •
edited
Loading

Ah no, wait, you're on Bazzite/Fedora 41 and you are attempting to use a vulkaninfo from CentOS Stream 9.

uname -a                                                                                                                                                                                                                               
Linux 6.12.12-203.bazzite.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb  4 23:54:25 UTC 2025 x86_64 GNU/Linux
[...]
$ curl -LO https://kojihub.stream.centos.org/kojifiles/packages/vulkan-tools/1.3.268.0/1.el9/i686/vulkan-tools-1.3.268.0-1.el9.i686.rpm

Please try again with the one from Fedora 41, I'm sure it will work. You can also install it directly on an x86_64 system:

$ sudo rpm -Uvh https://kojipkgs.fedoraproject.org//packages/vulkan-tools/1.4.304.0/1.fc41/i686/vulkan-tools-1.4.304.0-1.fc41.i686.rpm

@drelbszoomer
Copy link
Author

Ah no, wait, you're on Bazzite/Fedora 41 and you are attempting to use a vulkaninfo from CentOS Stream 9.

uname -a                                                                                                                                                                                                                               
Linux 6.12.12-203.bazzite.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb  4 23:54:25 UTC 2025 x86_64 GNU/Linux
[...]
$ curl -LO https://kojihub.stream.centos.org/kojifiles/packages/vulkan-tools/1.3.268.0/1.el9/i686/vulkan-tools-1.3.268.0-1.el9.i686.rpm

Please try again with the one from Fedora 41, I'm sure it will work. You can also install it directly on an x86_64 system:

$ sudo rpm -Uvh https://kojipkgs.fedoraproject.org//packages/vulkan-tools/1.4.304.0/1.fc41/i686/vulkan-tools-1.4.304.0-1.fc41.i686.rpm

Nope, same issue with that one:

LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for explicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/explicit_layer.d
LAYER:                   /etc/xdg/vulkan/explicit_layer.d
LAYER:                   /etc/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /usr/local/share/vulkan/explicit_layer.d
LAYER:                   /usr/share/vulkan/explicit_layer.d
LAYER:                Found no files
DRIVER:            Searching for driver manifest files
DRIVER:               In following locations:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
DRIVER:               Found the following files:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
DRIVER:            Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json, version 1.0.1
DEBUG | DRIVER:    Searching for ICD drivers named /usr/lib64/libGLX_nvidia.so.0
INFO:              /usr/lib64/libGLX_nvidia.so.0: wrong ELF class: ELFCLASS64
DRIVER:            Requested ICD /usr/lib64/libGLX_nvidia.so.0 was wrong bit-type. Ignoring this JSON
ERROR: [Loader Message] Code 0 : vkCreateInstance: Found no drivers!
ERROR | DRIVER:    vkCreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /builddir/build/BUILD/vulkan-tools-1.4.304.0-build/Vulkan-Tools-vulkan-sdk-1.4.304.0/vulkaninfo/./vulkaninfo.h:456:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

In fact I don't have this issue on Fedora

It's very possible there's something wrong with my build here (I'm using a bazzite/silverblue variant, and the dnf5 install cmd above is pulling from negativo17-fedora-nvidia and not negativo17-fedora-multimedia), but I'm at a loss as to what it is. The ICDs are there. The (correct) nvidia libs are there for both. i686 and x86_64 variants of vulkan-loader are there - and yet the i686 ICD simply isn't being discovered.

@scaronni
Copy link
Member

scaronni commented Feb 11, 2025 •
edited
Loading

This is strange:

INFO:              /usr/lib64/libGLX_nvidia.so.0: wrong ELF class: ELFCLASS64

Can you do a ldd /usr/lib*/libGLX_nvidia.so.0?

@scaronni
Copy link
Member

This is strange if that's the output of 32 bit vulkaninfo, it should not hit that file:

DRIVER:            Searching for driver manifest files
DRIVER:               In following locations:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
DRIVER:               Found the following files:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
DRIVER:            Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json, version 1.0.1

I can't reproduce it on my system.

@drelbszoomer
Copy link
Author

This is strange:

INFO:              /usr/lib64/libGLX_nvidia.so.0: wrong ELF class: ELFCLASS64

Can you do a ldd /usr/lib*/libGLX_nvidia.so.0?

ldd /usr/lib*/libGLX_nvidia.so.0                                                                                                                                                                                                       [11:12:46]
/usr/lib64/libGLX_nvidia.so.0:
	linux-vdso.so.1 (0x00007f9c40d13000)
	libnvidia-glsi.so.570.86.16 => /lib64/libnvidia-glsi.so.570.86.16 (0x00007f9c40b35000)
	libnvidia-tls.so.570.86.16 => /lib64/libnvidia-tls.so.570.86.16 (0x00007f9c40b2f000)
	libnvidia-glcore.so.570.86.16 => /lib64/libnvidia-glcore.so.570.86.16 (0x00007f9c3e800000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f9c3e6bb000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007f9c40b1b000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9c3e4c8000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f9c40b14000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9c40b0f000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f9c40b0a000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f9c3e3e2000)
	libnvidia-gpucomp.so.570.86.16 => /lib64/libnvidia-gpucomp.so.570.86.16 (0x00007f9c3a600000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f9c40adc000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9c40d15000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f9c40ad6000)
/usr/lib/libGLX_nvidia.so.0:
	linux-gate.so.1 (0xf7eeb000)
	libnvidia-glsi.so.570.86.16 => /lib/libnvidia-glsi.so.570.86.16 (0xf7cd8000)
	libnvidia-tls.so.570.86.16 => /lib/libnvidia-tls.so.570.86.16 (0xf7cd1000)
	libnvidia-glcore.so.570.86.16 => /lib/libnvidia-glcore.so.570.86.16 (0xf5e5d000)
	libX11.so.6 => /lib/libX11.so.6 (0xf5d08000)
	libXext.so.6 => /lib/libXext.so.6 (0xf5cf2000)
	libc.so.6 => /lib/libc.so.6 (0xf5af3000)
	libdl.so.2 => /lib/libdl.so.2 (0xf5aee000)
	libpthread.so.0 => /lib/libpthread.so.0 (0xf5ae9000)
	librt.so.1 => /lib/librt.so.1 (0xf5ae4000)
	libm.so.6 => /lib/libm.so.6 (0xf5a0a000)
	libnvidia-gpucomp.so.570.86.16 => /lib/libnvidia-gpucomp.so.570.86.16 (0xf0333000)
	libxcb.so.1 => /lib/libxcb.so.1 (0xf0306000)
	/lib/ld-linux.so.2 (0xf7eed000)
	libXau.so.6 => /lib/libXau.so.6 (0xf0301000)

which is what I'd expect - the 32 bit libraries are there, but the loader is only trying to load the one from /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json and is ignoring /usr/share/vulkan/icd.d/nvidia_icd.i686.json

@drelbszoomer
Copy link
Author

Ah, interestingly if I force the i686 loader thru VK_DRIVER_FILES, it correctly loads the 32-bit drivers/ICD:

VK_LOADER_DEBUG=all VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.i686.json  ./vulkaninfo32

INFO:              Vulkan Loader Version 1.4.304
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for explicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/explicit_layer.d
LAYER:                   /etc/xdg/vulkan/explicit_layer.d
LAYER:                   /etc/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /usr/local/share/vulkan/explicit_layer.d
LAYER:                   /usr/share/vulkan/explicit_layer.d
LAYER:                Found no files
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for explicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/explicit_layer.d
LAYER:                   /etc/xdg/vulkan/explicit_layer.d
LAYER:                   /etc/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /usr/local/share/vulkan/explicit_layer.d
LAYER:                   /usr/share/vulkan/explicit_layer.d
LAYER:                Found no files
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for explicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/explicit_layer.d
LAYER:                   /etc/xdg/vulkan/explicit_layer.d
LAYER:                   /etc/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /usr/local/share/vulkan/explicit_layer.d
LAYER:                   /usr/share/vulkan/explicit_layer.d
LAYER:                Found no files
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
DRIVER:            Searching for driver manifest files
DRIVER:               In following locations:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:               Found the following files:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:            Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.i686.json, version 1.0.1
DEBUG | DRIVER:    Searching for ICD drivers named /usr/lib/libGLX_nvidia.so.0
DRIVER:            Searching for driver manifest files
DRIVER:               In following locations:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:               Found the following files:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:            Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.i686.json, version 1.0.1
DEBUG | DRIVER:    Searching for ICD drivers named /usr/lib/libGLX_nvidia.so.0
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
INFO:              No valid vk_loader_settings.json file found, no loader settings will be active
INFO:              Portability enumeration bit was set, enumerating portability drivers.
LAYER:             Searching for implicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/implicit_layer.d
LAYER:                   /etc/xdg/vulkan/implicit_layer.d
LAYER:                   /etc/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/implicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                   /usr/local/share/vulkan/implicit_layer.d
LAYER:                   /usr/share/vulkan/implicit_layer.d
LAYER:                Found the following files:
LAYER:                   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file version 1.0.0)
INFO:              Found manifest file /usr/share/vulkan/implicit_layer.d/nvidia_layers.json (file version 1.0.1)
LAYER:             Searching for explicit layer manifest files
LAYER:                In following locations:
LAYER:                   /var/home/bleggett/.config/vulkan/explicit_layer.d
LAYER:                   /etc/xdg/vulkan/explicit_layer.d
LAYER:                   /etc/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/vulkan/explicit_layer.d
LAYER:                   /var/home/bleggett/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                   /usr/local/share/vulkan/explicit_layer.d
LAYER:                   /usr/share/vulkan/explicit_layer.d
LAYER:                Found no files
DRIVER:            Searching for driver manifest files
DRIVER:               In following locations:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:               Found the following files:
DRIVER:                  /usr/share/vulkan/icd.d/nvidia_icd.i686.json
DRIVER:            Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.i686.json, version 1.0.1
DEBUG | DRIVER:    Searching for ICD drivers named /usr/lib/libGLX_nvidia.so.0
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
WARNING | LAYER:   Layer VK_LAYER_MESA_device_select uses API version 1.3 which is older than the application specified API version of 1.4. May cause issues.
ERROR: [Loader Message] Code 0 : libVkLayer_MESA_device_select.so: cannot open shared object file: No such file or directory
ERROR:             libVkLayer_MESA_device_select.so: cannot open shared object file: No such file or directory
INFO | LAYER:      Requested layer "VK_LAYER_MESA_device_select" failed to load.
LAYER:             vkCreateInstance layer callstack setup to:
LAYER:                <Application>
LAYER:                  ||
LAYER:                <Loader>
LAYER:                  ||
LAYER:                <Drivers>
INFO | DRIVER:     linux_read_sorted_physical_devices:
INFO | DRIVER:          Original order:
INFO | DRIVER:                [0] NVIDIA GeForce RTX 2060
INFO | DRIVER:          Sorted order:
INFO | DRIVER:                [0] NVIDIA GeForce RTX 2060  
DRIVER | LAYER:    vkCreateDevice layer callstack setup to:
DRIVER | LAYER:       <Application>
DRIVER | LAYER:         ||
DRIVER | LAYER:       <Loader>
DRIVER | LAYER:         ||
DRIVER | LAYER:       <Device>
DRIVER | LAYER:           Using "NVIDIA GeForce RTX 2060" with driver: "/usr/lib/libGLX_nvidia.so.0"
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.304


Instance Extensions: count = 24
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 2
--------------------------
VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1
VK_LAYER_NV_optimus         NVIDIA Optimus layer         1.4.303  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.303
	driverVersion      = 570.86.16.0
	vendorID           = 0x10de
	deviceID           = 0x1e89
	deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
	deviceName         = NVIDIA GeForce RTX 2060
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 570.86.16
	conformanceVersion = 1.4.1.0
	deviceUUID         = 925d432e-5317-d686-9fd3-93fe18cf833d
	driverUUID         = 5429646d-3d95-54e1-b807-75d58fc2ef85

So it does appear to be an ICD path thing, if I don't force it, the loader won't find nvidia_icd.i686.json

@drelbszoomer
Copy link
Author

drelbszoomer commented Feb 13, 2025 •
edited
Loading

Oh good grief.

export VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.x86_64.json was in my ~/.zshenv

I don't remember adding that at all, but that is (obviously) the problem and that (obviously) fixed it.

PEBKAC. Apologies for the noise. TYVM for the work you do.

@scaronni
Copy link
Member

😄

No worries, it's good to check if we're not sure. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@scaronni @drelbszoomer