Discoloration/washed out colors - nVidia

[SveSop]

I am struggling a bit with what i see as "washed out colors" when using DXVK.

I dont know if these issues is due to the "have to hack-fix with spirv-opt" shader compilation on the nVidia drivers, or if its a nVidia issue.. or just me.
If you take a look at the 2 attached pictures, its a side-by-side comparison from Unigine Valley, and Unigine Superposition. Its clearly a lot brighter or "bland" in Valley, and in Unigine Superposition it almost looks like its a thin layer of dust covering everything.

Just me?
nVidia GTX 970 w/ 390.25 driver
wine-vulkan-3.2

[doitsujin]

Please attach log files, it seems that some of the shaders aren't working (or at least not doing the right thing). Valley renders correctly on RADV.

[pchome]

With staging-vulkan looks ok for me (with opt hacks too).
wine-3.2+staging-vulkan+nVidia GTX 750 Ti w/ 387.34 driver

BTW you don't need to have full wine-staging installed but only vulkan-1.dll.so and vulkan.dll.so somewhere wine can find them.
Me using ~/.local/lib64/wine and export WINEDLLPATH=${HOME}/.local/lib64/wine in my ~/.profile.

[SveSop]

@pchome What is "staging-vulkan"? Wine-vulkan https://github.com/roderickc/wine-vulkan is not really "staging" if that is what you mean?

Wine-vulkan is "vulkan-loader" patched ontop of wine-3.2, and if modifying wine-dev = staging, then i understand what you mean. I just mention this so there is no missunderstanding between wine-staging with staging-dev's implementation of vulkan api vs. wine-vulkan-3.2 tho.

[pchome]

What is "staging-vulkan"?

dlls/vulkan and dlls/vulkan-1 from wine-staging-2.21 compiled separately.
https://github.com/pchome/dxvk/tree/winebuild/external/vulkan - there is vulkan-init.sh if you want to try.

$ sh vulkan-init.sh to retrieve all needed files and than regular meson magic.
Or simply grab vulkan-1.dll.so and vulkan.dll.so (fakedlls/vulkan-1.dll and fakedlls/vulkan.dll (?)) files from wine-staging-2.21 .

[SveSop]

@pchome Oki :) So, in principle not that really changed vs. using wine-vulkan. Also tested this with wine-staging-2.21, and there was no change. (Did not think it would be either, afaik its just a loader for the linux vulkan api, and not a "driver" in itself). I could ofc be totally wrong, and the problem lies with my linux vulkan api's tho. Gonna do some searching to see if i can find a native vulkan test that also have a windows vulkan demo. nVidia have some, but they do not run too well under wine.

@doitsujin Attaching logs from superposition and valley, aswell as screenshots comparing wine-dx11 vs wine-dxvk. The wine-dx11 of superposition is severely bugged i guess due to some missing functions or something, but valley runs well.
superposition_d3d11.log
superposition_dxgi.log
superposition_vulkanlog.txt

Valley_d3d11.log
Valley_dxgi.log
Valley_vulkanlog.txt

[pchome]

@SveSop

So, in principle not that really changed vs. using wine-vulkan.

Who knows.

At the moment the patchset provides Vulkan 1.0.51 core with minimal extensions for graphics rendering.

vs

Vulkan API version: 1.0.56
SDK 1.0.61.1
driver 387.34

[SveSop]

@pchome Well.. i dont know. Still not working for me.
After a funny trip to hell - yeah, 387.34 does NOT play well with 4.15 kernel... i ended up trying 384.111 from ubuntu repo's, and still same problem.

What libvulkan1 libraries are you using on your linux distro? and what distro for that matter? :)

[SveSop]

I have run the "ThreadedRenderingVK" fish-demo from nvidia, and the colors seem fine. I also ran vkQuake for windows in wine (using Vulkan), and the colors i guess seem oki, but the game is from the ages, so there is not really any fancy effects to view.

The problem i find is that there is not really that many pre-compiled binaries for Linux to test when it comes to Vulkan stuff... They all requires a random number of different compilers and libraries.. and other versions of whatever, so i go stir crazy just trying to compile some crap demo.

Since noone else seem to have any problems with DXVK and the colors, it's probably something i do wrong when i compile or something.. i dunno. Thinking of taking a couple of years break, so this thing perhaps gets in some distro's repository for me to test :)

[pchome]

What libvulkan1 libraries are you using on your linux distro?

Gentoo

$ equery b libvulkan.so
 * Searching for libvulkan.so ...
media-libs/vulkan-loader-1.0.61.1 (/usr/lib/libvulkan.so -> libvulkan.so.1)
media-libs/vulkan-loader-1.0.61.1 (/usr/lib64/libvulkan.so -> libvulkan.so.1)

$ ls -1 /usr/lib64/wine-staging-2.21/wine/vulkan* 
/usr/lib64/wine-staging-2.21/wine/vulkan-1.dll.so
/usr/lib64/wine-staging-2.21/wine/vulkan.dll.so

[SveSop]

$ dpkg -l|grep libvulkan1
ii  libvulkan1:amd64  1.0.61.1+dfsg1-1ubuntu1  amd64        Vulkan loader library
ii  libvulkan1:i386  1.0.61.1+dfsg1-1ubuntu1  i386         Vulkan loader library

Using wine-vulkan-3.2 i seem to only get washed-out colors with dxvk. Same with wine-staging-2.21.
Have LunarG SDK 1.0.51 installed with icd loader in my wineprefix.

I think i will have to conclude with me doing something wrong when compiling dxvk, so ill fiddle around with it a bit more. Tips are welcome :)

[pingubot]

@SveSop , i have the same issue here with superposition. using the same driver and a gtx 970. will check with valley

[SveSop]

@pchome

Vulkan API version: 1.0.56

Has not really much to do with it when viewing it in gpu caps viewer, cos thats from the driver. I have 1.0.65 due to nVidia 390.25 i guess. Atleast what is showing in caps viewer. I did not check what 384.111 had tho im afraid.

@pingubot Hmm.. interesting. I had problems installing the 387.34 driver that @pchome is using cos of my 4.15 kernel.. and i believe some retpoline patches the kernel source has that does not work too well when compiling with DKMS even on 4.4 current kernel for Ubuntu. I guess Gentoo dont have binary drivers, so is the driver is compiled from nVidia source downloaded from nVidia.com?

[pchome]

I guess Gentoo dont have binary drivers, so is the driver is compiled from nVidia source downloaded from nVidia.com?

Yes, and patched w/ 4.15-rc1-384.98.patch.gz

[SveSop]

Ill maybe have a peek at the patches if its possible to use this on the Ubuntu 387.34 DKMS source.. cos if not, i wont fiddle with the nVidia source drivers as this usually tends to go tits up if not done right under Ubuntu :)

There is a slight discrepency with kernel 4.15 and not using gcc-7.3 for the "wanted" retpoline gcc functions when compiling binary drivers. So far no gcc-7.3 available at the Toolchain-test PPA, and no backport for gcc-8 for Ubuntu.

[SveSop]

Oh, and if possible @pchome could you post a side-by-side comparison picture of a scene in Vally between wine-devel-3.2 w/dx11 and DXVK w/vulkan? (Put up DXVK hud too if you can)

[SveSop]

@pchome

1. Have tested libvulkan1 1.0.51 - No change
2. Have tested libvulkan1 1.0.61 - No change
3. Compiled my own libvulkan1 1.0.69 from git - No change
4. Have tested pre-built dxvk binaries (from some link posted here in another thread - build new each commit.) - No change
5. Have tested wine-vulkan & wine-staging-2.21 - No change
6. Added the patch from the link you posted to my DKMS and downgraded to 387.34 - No change

The only thing - save from installing Gentoo - i have not tested is that staging-vulkan dll.o loader compiled you have mentioned.

Could you maybe post a screenshot of the "desk" in Superposition just for comparison?

[pchome]

[SveSop]

@pchome
Interesting. I wonder what makes the change. Some weird Ubuntu/Debian problem?

[SveSop]

Unless ofc the 9xx model is "vulkan bugged" when it comes to drivers? @pingubot also run a 970 if i remember correct...

[pingubot]

yes, i also have a970. I also tried downgrading the drivers to 384,111, but the issue persists. In addition to the wrong colours, i have the issue that the benchmark is heavily stuttering (the shown fps do not reflect that).

[SveSop]

@pingubot I think that fps-stutter is some sort of wine problem, cos that happens with wine if you run opengl too. It is not like that in the "Game" mode tho, so not sure what's going on there.

@doitsujin I tested one more thing in my search for this problem.. and that was that i bought "The Talos Principle" on steam.
Installed steam in wine, and installed "The Talos Principle". Tested both with vulkan and dxvk, and there does not seem to be any rendering problems there. Neither did i have to use the DXVK_SHADER_OPTIMIZE option either, as there was seemingly no problems switching to DirectX11 mode with dxvk (Had the DXVK_HUD showing what it was supposed to, but it does not show on the games screenshot function)
I also tried to run the game with DirectX11 without DXVK, but had loads of graphics annomalies then, and was not really able to play.
I will investigate a bit more, and see if i can find very lit areas and make some more comparisons, but so far it does not look horrible. Also attached d3d11 and dxgi logs.

Talos Principle is obviously another graphics engine than Unigine tho.. And from what i have seen so far, Superposition uses "Unigine 2", and Heaven/Valley uses "Unigine 1".. something like that. The discoloration also seems far worse with Superposition and "Unigine 2" engine.

Screenshots attached: Vulkan_wine

Vulkan_Linux:

OpenGL_Linux:

dxvk_wine:

Talos_d3d11.log
Talos_dxgi.log

[SveSop]

Example of what i believe has to do with some sort of "surface reflection" problem with Unigine Superposition. In the back of the scene, there is some lightswitches, and turning those on creates reflection.. like you see on the OpenGL screenshots (Linux native).

On the DXVK versions, you see that the "red" color which kinda shines from the device behind your back is overly reflecting on everything, and turning the light on makes the light reflect waaay too much.

Finding what reflection function in the Unigine2 Engine that bugs out vs. lightreflections in the Talos Principle that works i have no clue how to start pondering.. i leave that up to those with greater ideas than me :)

OpenGL Native - Light off, then light on:


Then with DXVK and wine - Light off, then light on:

[doitsujin]

If you actually look at the log files, Superposition is not even supposed to work - it requires Stream Output, which isn't implemented yet.

As for Valley, it fails to compile one particular pipeline, which is probably responsible for parts of the lighting (which is probably caused by another bug in the shader compiler):

debug: Compiling graphics pipeline...
debug:   vs  : VS_b759414d4739d2532c5f94cdd008c1233f0fd9be
debug:   fs  : PS_2dbb78a2cc3f5b6a3c423b5b068de5f47ccc5217
err:   DxvkGraphicsPipeline: Failed to compile pipeline

There is no "washed out colors" problem specifically, even though the symptoms might suggest there is.

[SveSop]

@doitsujin

There is no "washed out colors" problem specifically, even though the symptoms might suggest there is.

Agree. Would it perhaps be better to call it "texture reflection" problem?

[pdaniell-nv]

@pchome Do you see any change if you use 390.25 with your GTX 750?

[pdaniell-nv]

@doitsujin Can you give me a pointer to the shader that fails to compile. I'll take a look here to see what's causing it to fail. Thanks.

[pchome]

@pdaniell-nv
ok, I'll try to upgrade and report about changes
but with 390.12/390.25 I met most issues from recently discussed topics on https://devtalk.nvidia.com/default/board/98/linux/

[ZeroFault]

@pdaniell-nv

Here are the shaders bound to the pipeline when compiling fails.
failingshaders.zip
I don't see any vulkan validation errors when it fails either.

[pchome]

@pdaniell-nv
No noticeable changes so far.

[pdaniell-nv]

@pchome Thanks for the update. So the 970 vs 750 issue doesn't appear to be driver related, but GPU. I still have no brilliant ideas why that might be the case, especially after the device simulation that @SveSop did. I tried to reproduce it locally but on Windows superposition crashed on startup when using the dxvk binaries from 2/24. I didn't dig into much further. Perhaps I'll try heaven next.

[pchome]

@pdaniell-nv

I tried to reproduce it locally but on Windows superposition crashed on startup when using the dxvk binaries from 2/24.

If this related to launcher bugs it's possible to run benchmark directly with corresponding parameters passed:

superposition.exe -preset 0 -video_app direct3d11 -video_fullscreen 2 -shaders_quality 1 -textures_quality 1 -dof 1 -motion_blur 1 -video_vsync 0 -video_mode -1 -console_command "world_load superposition/superposition && render_manager_create_textures 1" -project_name Superposition -video_width 1920 -video_height 1080 -extern_plugin GPUMonitor -mode 0 -sound 1 -tooltips 1

It's looks like -preset 0 = Custom , -video_fullscreen 2 = Borderless window and -mode 0 = Benchmark

@ all
Also for those wondering there is comparison mode on http://vulkan.gpuinfo.org
970 vs 750Ti
Intel HD 530 vs 970 vs 750Ti vs Vega
Maybe this information would be helpful.

[SveSop]

@pdaniell-nv

I tried to reproduce it locally but on Windows superposition crashed on startup when using the dxvk binaries from 2/24. I didn't dig into much further. Perhaps I'll try heaven next.

Hopefully you remembered to add the env variable: DXVK_SHADER_OPTIMIZE=1 :)

[SveSop]

@doitsujin @pchome
Tested with that commit, and it seems as it does not crash with that. Was able to use just a wee bit over 4GB out of 4039MB available (according to nVidia-smi), and even tho i selected higher shaders, nothing happened.
Now, if i start it with all set to max (Resolution 8K, shadersQ=8K, Texture=High) it just loaded, and well.. i did not bother waiting long enough cos it seemingly stalled or atleast did not complete the load of the scene. Starting it with Res=8K, ShaderQ=8K, Texture=low indicate 3850MB vram usage (from Linux version), and swithing texture quality AFTER "game mode" was started did the same as in wine - max out at a tad over 4GB vmem usage.

As i mentioned earlier - starting the linux version with > 4GB vram usage for me, throws a warning message, but in wine it will not due to not able to determine how much vram i actually have.

So i guess the crashing is gone, or atleast heavily improved.Lets see what results @pchome gets.

[SveSop]

Did some more testing, and a "debug" log running tail -f of the file, and it seems as when i load >4GB of vram, the graphics pipeline compiling just stops and nothing more happens and i have to just kill wine.

I also observed VRAM usage max out around 4GB on my 4GB card, so instead of crashing, it just loads textures to vram until "full", and then stops doing anything useful. No observed increase in system ram usage either...

[pdaniell-nv]

@SveSop @pchome Thanks for the tips. I was able to run it locally and I can see what the issue with Superposition probably is with NVIDIA GPUs. There are two shaders that are using 14 uniform buffers, but our limit for maxPerStageDescriptorUniformBuffers is 12. In this case there are probably two constant buffers for each shader that are getting dropped on the floor. If possible you could spill uniform buffers over 12 to use SSBOs instead. The validation layers should have caught this error. If they're not, it's probably worth filing an issue here: https://github.com/KhronosGroup/Vulkan-LoaderAndValidationLayers/issues.

[SveSop]

@doitsujin From looking at the hardware database for eg. a AMD RX480 card, it shows
maxPerStageDescriptorUniformBuffers | 4294970000
http://vulkan.gpuinfo.org/displayreport.php?id=1028#limits
Is that something that can be tweaked/limited in the dxbc/spirv compiler you think?

[doitsujin]

I'll be honest, I did not expect this. The code was built around the assumption that Vulkan drivers on D3D11-capable hardware would at least provide support for the features and limits that D3D11 requires.

On AMD, the number of descriptors is practically unlimited, and even Intel drivers seem to report fairly generous limits in that regard. @pdaniell-nv is this a hardware limitation or is there a chance that this limit may be increased in future drivers?

Using read-only SSBOs instead of uniform buffers is certainly possible, but quite a lot of work, and it needs to be tested carefully.

[pdaniell-nv]

@doitsujin The reason our UBO limit for Vulkan is less than DX and OpenGL is that we have to use some constant buffers internally in Vulkan as part of our implementation. Unfortunately it's not something we can easily change. If we were to match DX it looks like we would need at least 15 UBOs (D3D11_COMMONSHADER_CONSTANT_BUFFER_HW_SLOT_COUNT).

How many descriptor sets does DXVK use?

[doitsujin]

@pdaniell-nv DXVK would need 14 UBO slots since that's what D3D11 applications are allowed to use at most (D3D11_COMMONSHADER_CONSTANT_BUFFER_API_SLOT_COUNT is the relevant limit here). I believe the 15th slot is reserved for the driver, but DXVK does not actually make use of that.

How many descriptor sets does DXVK use?

Each pipeline layout consists of only one descriptor set layout, if that's what you're asking, with one VkDescriptorSet per draw call in the worst case.

[pingubot]

any news on that topic @pdaniell-nv ?

[pdaniell-nv]

Yes, we are investigating increasing our UBO limit. If successful, a driver with the new limit will be available mid-April.

[pingubot]

@pdaniell-nv , thank you for the information,Hopefully you can get it implemented. Btw, beside the ubo limit, do you have a timeframe when all the fixes in the nvidia shader compiler will land in newer drivers ? I installed 390.42, are fixes already in there ?

[pdaniell-nv]

The BitfieldInsert/BitfieldExtract bug should be fixed with Linux driver 390.42. The other bugs are fixed in later drivers.

[pdaniell-nv]

We just released a new 396.18.02 beta driver for Linux at https://developer.nvidia.com/vulkan-driver. This new driver increases the per-shader stage limit for UBOs from 12 to 15. Hopefully this fixes Superposition.

[SveSop]

This is as close as it can get come colors it seems @pdaniell-nv . Have not done any benching or "flip lightswitches" yet, but so far an ocean of difference :) ("Native" being Linux OGL client)
For comparison: (with build: 8eb7859)

[doitsujin]

Great, and thanks for reporting back @SveSop - looks like I can close this then.