*Failed to start recording* *Failed to open NVENC codec: Generic error in an external library*

[Benjamin-Loison]

on work laptop on Debian 12 GNOME.

While the latter software seems not Linux compatible, what about the former? They should not be mentioned in my case.

nvidia-smi

Output:

Wed Dec 11 15:20:50 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A1000 6GB Lap...    On  | 00000000:01:00.0  On |                  N/A |
| N/A   51C    P5               6W /  45W |    160MiB /  6144MiB |     33%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2851      G   /usr/lib/xorg/Xorg                           69MiB |
|    0   N/A  N/A      3035    C+G   ...libexec/gnome-remote-desktop-daemon       81MiB |
+---------------------------------------------------------------------------------------+

Settings > Output > Recording > Encoder:

• Hardware (NVENC, H.264) triggers the mentioned issue
• Hardware (NVENC, HEVC) triggers the mentioned issue
• Software (x264) works fine

Hardware may be about leveraging the NVIDIA GPU.

Related to #23.

[Benjamin-Loison]

Related to Benjamin_Loison/Ubuntu/issues/16.

[Benjamin-Loison]

Hardware (NVENC, H.264) would be a good start.

[Benjamin-Loison]

DuckDuckGo search Linux "Failed to open NVENC codec: Generic error in an external library".

[Benjamin-Loison]

sudo apt install -y libffmpeg-nvenc-dev

Output:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  libffmpeg-nvenc-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 60.5 kB of archives.
After this operation, 392 kB of additional disk space will be used.
Get:1 http://ftp.fr.debian.org/debian bookworm/main amd64 libffmpeg-nvenc-dev all 11.1.5.2-1 [60.5 kB]
Fetched 60.5 kB in 0s (749 kB/s)               
Selecting previously unselected package libffmpeg-nvenc-dev.
(Reading database ... 361364 files and directories currently installed.)
Preparing to unpack .../libffmpeg-nvenc-dev_11.1.5.2-1_all.deb ...
Unpacking libffmpeg-nvenc-dev (11.1.5.2-1) ...
Setting up libffmpeg-nvenc-dev (11.1.5.2-1) ...

Source: https://www.reddit.com/r/obs/comments/wo9z2e/comment/kprvb6o/

[Benjamin-Loison]

https://www.reddit.com/r/obs/comments/wo9z2e/comment/kpwfze8/ does not help.

[Benjamin-Loison]

sudo apt update
sudo apt upgrade -y
sudo apt autoremove -y

seems to state that the laptop is up-to-date.

Maybe restarting help.

[Benjamin-Loison]

After restart the problem is solved. nvidia-smi shows obs during recording.

[Benjamin-Loison]

Maybe disabling secure boot involves again this issue, see Benjamin-Loison/cinnamon/issues/178. However, nvidia-smi still works fine.

[Benjamin-Loison]

Let us try to temporarily reenable secure boot to verify this behavior. It works fine if I reenable secure boot.

[Benjamin-Loison]

Maybe have to regenerate initramfs, kernel module or/and grub.

[Benjamin-Loison]

I disabled again secure boot to leverage hibernation which seems more important to me as there is a CPU alternative for OBS.

[Benjamin-Loison]

DuckDuckGo and Google search "Generic error in an external library" after disabling "secure boot".

[Benjamin-Loison]

Only relevant results seem to be:

• https://www.reddit.com/r/debian/comments/1cvz3qb/comment/l4tbl4c/
• NVIDIA/open-gpu-kernel-modules/issues/104#issuecomment-1246276388
• https://www.reddit.com/r/debian/comments/1bwroju/virtual_box_errors_in_debian_sid_unstable/

I should investigate them.

[Benjamin-Loison]

Well I was still having the issue with secure boot enabled if I remember correctly, so I disabled again secure boot in addition to sudo update-initramfs -u -k all in the context of https://help.ubuntu.com/community/EnableHibernateWithEncryptedSwap. Hibernation works fine as well, so I have the best of both worlds for an unknown reason. As always NVIDIA and secure boot seem to blame.

[Benjamin-Loison]

Maybe related to Benjamin_Loison/Debian/issues/129.