NV_ENC_DEVICE_TYPE_DIRECTX support

[SoftDevelIM]

Hello,

My question is: ...
Is there a way to make it work with NVENC sessions using device NV_ENC_DEVICE_TYPE_DIRECTX
It was working until driver 440.97 but now it has stopped working, using this path once no more increases the session limit also for DirectX sessions. ¿Could it be done somehow?

[Snawoot]

Hello,

Try to apply new x86 library patch as well. If it doesn't work, use one of session bump methods.

If none of this will work, please provide simple binary which tries to open 3 DirectX encoding sessions and outputs whether it succeeded or not - it'll help me to reproduce this issue on my testbed. Plain console application will do.

[Snawoot]

P.S. x64 patch has to be applied to %WINDIR%\system32\nvcuvid.dll and x32 patch has to be applied to %WINDIR%\SysWOW64\nvcuvid.dll. Here is updated instructions.

[SoftDevelIM]

Yeah, patch is properly installed, let me see if i can create a sample that failes creating NV_ENC_DEVICE_TYPE_DIRECTX sessions

[SoftDevelIM]

I have the simple console app you can use to test here:
https://wetransfer.com/downloads/62008e3e914777dfc4ec686f9b57403f20191024163158/53c80c60076cfab4643217622e57319f20191024163158/e571bc

[Snawoot]

@SoftDevelIM Thanks! Received.

[Snawoot]

Here is my findings on this subject.

Since latest nvidia driver (440.97) session bumping technique doesn't work anymore, unlike it used to be before.

Only remaining way to overcome this is issue is to patch nvwgf2umx.dll driver component. I've managed to make such patch for version 440.97, but I didn't tested if real encoding session works because test binary doesn't have actual encoding capability.

There is also more important problem with it. nvwgf2umx.dll driver component is covered with code signing and we can't just change some bytes in binary image file. For testing purposes I did changes in process memory with my debugger. One can apply my resulting patch with 1337patch tool to running process with simple script like this. However, this solution (even if it works - it's remains unclear yet) is far from one which we can propose to patch users.

Unfortunately, I don't see appropriate workaround for driver code signing. Test mode is probably not what users want and also we can't distribute patched binaries (it's some kind of policy). I hope someone can propose acceptable solution, because I'm not very familiar with Windows driver signing.

[SoftDevelIM]

I am trying to find this dll into my process when running and encoding and I can only see nvldumxdx.dll and nvwgf2umx_cfg.dll, is it somehow renamed¿?

[Snawoot]

No, it's loaded dynamically during encoder initialization and invoked from DirectX code. You probably should set break on module load or set breakpoint somewhere near encoder initialization.

[Snawoot]

Also I found similar patch which performs signed binary patching. It requires user to install Driver Development Kit and put system into test mode. There is powershell script available to do actual patching and code signing. If someone can adapt this recipe to our case, then we probably will be able to carry out full persistent patching of DirectX encoder.

@sk1080 Could you help us with that?

[SoftDevelIM]

It is strange becouse in the sample I made I can clearly see with Process Explorer:
Loaded into the example:
C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_67c16e22f0dab361\nvwgf2umx.dll

Loaded into my app:
C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_67c16e22f0dab361\nvwgf2umx_cfg.dll

I cant find nvwgf2umx.dll into my production process :/

[Snawoot]

@SoftDevelIM in my observation nvwgf2umx.dll gets loaded long after nvwgf2umx_cfg.dll. Try to set breakpoint on NvEncodeAPICreateInstance of nvencodeapi64.dll - I see nvwgf2umx.dll loaded at this point in your example code.

[Snawoot]

Here is search pattern and replacement for those who would like to try to adapt @sk1080 patcher script for this case.

Search binary string: 000F95C084C0741B
Replacement binary string: 000F95C00C01741B
Module name: nvwgf2umx.dll

I'm lost at handling all those manipulations in script.

[Snawoot]

I've adjusted powershell script to patch nvwgf2umx.dll and sign it: https://github.com/Snawoot/nvidia-patch/blob/win_dx/win/win10_x64/440.97/nvwgf2umx-patch.ps1

However, no luck. Encoder doesn't initializes at all with self-signed dll, throwing some exception. I think there is not much I can do about that.

[SoftDevelIM]

I am investigating why is nvwgf2umx_cfg.dll loaded instead of nvwgf2umx.dll to test your patch becouse I cant see it being loaded at any moment into my production app.

[SoftDevelIM]

No way I can understand the difference, my release app is an electron app using also C++ code and the sample app I sent to you is just a simple console application using some parts of the release app.

[oysteinkrog]

I reversed and found this working patch:
"nvencodeapi64.dll", "8BF085C0750549892FEB21", "33C08BF0750549892FEB21"

[Snawoot]

@oysteinkrog This is awesome! Thank you!

I can confirm this patch alone raises limits for CUDA NVENC in 64-bit ffmpeg, 32-bit ffmpeg and DX3D NVENC limit in nvencoders-test-x64.exe, provided by @SoftDevelIM, but not for nvencoders-test-Win32.exe, unfortunately. Maybe @oysteinkrog could provide patch for 32bit library as well?

This new patch already covers all cases covered by current patches applied to nvcuvid.dll and also 64bit D3D11 case, hence it superseeds current patch set.

I did only shallow testing with single mp4 video converted into 3 distinct resolutions simultaneously and it seems working well. It's still unclear if new patch has some unwanted side effects, so we need more testing from real users and if everything is right we're going to ship this new patch to end users instead of current one. Maybe we can even apply it to older versions of drivers.

@SoftDevelIM What is your current driver version? I'll put together .1337 file for your version so you can test whether patch is good for your app.

[Snawoot]

Reuploaded D3D11 test binaries (x64 and Win32) since original link expired: https://gist.github.com/Snawoot/5f91a00656a809b06ae14c2619a01b7a/raw/cc71bb26b72433c3eee5f570a23da68dfda24d5f/directx-to-nvencoders-test.rar

[Snawoot]

I think magic strings for 32bit SysWOW64/nvencodeapi.dll is 89450885C075048937EB22 -> 33C089450875048937EB22. Only testing part remains.

[Snawoot]

Solution delivered to master via PR #196