0.11.0 release - FFMPEG hardware accelleration not working inside docker container on i7-11th gen RocketLake #3941
Comments
|
We've had a lot of testing done with 12th gen and other recent intel hardware acceleration so the iHD driver should be working. A few things:
In both cases please provide the error logs if there are some 👍 |
|
-c:v h264_qsv by itself produces the error: The compose file currently: version: "3.3"
services:
frigate:
container_name: frigate
privileged: true
#restart: no
image: blakeblackshear/frigate:0.11.0
#image: blakeblackshear/frigate:stable-amd64
#image: blakeblackshear/frigate:stable-amd64nvidia
#runtime: nvidia
environment:
#- LIBVA_DRIVER_NAME=i965
#- LIBVA_DRIVER_NAME=iHD
shm_size: "4G"
devices:
- /dev/apex_0:/dev/apex_0
- /dev/dri/renderD128
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/opt/frigate/config.yml:/config/config.yml:ro
- /var/opt/frigate/data:/db
- /cctv:/media/frigate
- type: tmpfs
target: /tmp/cache
network_mode: host
environment:
FRIGATE_RPASS: "???"When trying with The error: |
it is important to note that on Frigate 0.10 which used ffmpeg 4.3 there was an unfortunate behavior where if a user requested hwaccel but it was not available ffmpeg would silently ignore that and use the CPU. We had a user without any iGPU claim it was working due to this, so it makes it difficult to know for sure if it was previously working. How does frigate's CPU usage with 0.11 compare with 0.10? Can you also go on your host and run |
|
also what is your host OS? |
|
Not to worry, this isn't my first rodeo, hardware accel works exceptionally well in 0.10.1 output of ls /dev/dri -lrt Host OS: |
|
That's unfortunate it is not working here :/ We have some docs on mapping a custom build of ffmpeg to replace the default one: https://docs.frigate.video/configuration/advanced#custom-ffmpeg-build We use BtBn by default, I wonder if you'll have luck using the 4.4 version instead of the 5.1 that we use by default: https://github.com/BtbN/FFmpeg-Builds/releases/tag/autobuild-2022-09-25-12-38 |
|
Just tried v4.4 of the static ffmpeg build, same issue, hardware accel inside the container not working. However if I use that static 4.4 build outside the container, works just fine. Is it possible to do what I just did with the 4.4 ffmpeg static build above, but instead with the one that was released with Frigate 0.10.1? any pointers? because that one most definitely works very well. |
|
At this point I don't think it is the ffmpeg version that matters (as you said it BtBn 4.4 works on your host). It seems more likely that the issue is the drivers we are installing. Perhaps there is an issue with that, you could try getting a shell in the container and upgrading apt and perhaps it'll pull updated drivers or there is some other driver needed for your iGPU 🤔 By the way in 0.10 we just got from the standard |
|
Yes that was one of the first things I attempted as in container updating via the container root shell. Well there is most definitely some issue here, I guess things will be realised more when and if a larger crowd upgrade to this version. I guess it was a wise idea to not put this on the stable-amd64 tag. |
To be clear, frigate is already out as |
|
I'll have to do some more looking and see if I can think of other things to try. |
|
You said you have many cameras, I'm curious what happens if you try hwaccel with just one using |
|
I do a manual test first in the container shell to confirm, before updating appropriate configuration string and restarting Frigate. So that has already been done, and didn't work. One thing I noticed is that there seems to be a mismatch between the iHD driver on the host and the one vainfo is reporting inside the container.... appears the Frigate version is older by a few revisions at least... |
Yeah that sounds right, could definitely be the issue. Not sure if there is a way to map the driver into the container and overwrite it. I believe our Dockerfile is pulling the most recent available driver for debian:11 |
|
@blakeblackshear May have some other ideas as well |
|
On 0.10.1 I use the args:
and it hardware decodes nicely: Also 0.10.1 is on the same iHD driver version as in 0.11.0. The FFMPEG in the 0.10.1 container is v4.3.1 |
|
You didn't share your config, so I just want to make sure of a few things:
|
|
Yes. I can disable hardware-acceleration, and 0.11.0 runs fine with no issues, recording and eventing etc. |
|
Without hwaccel enabled in 0.11 the cameras work right? Seems #3858 is likely related, might be worth trying #3858 (comment) |
|
Interestingly from that thread it seems his host driver is more recent than the container's too, in-fact, the exact same 'versions' as mine here. Ok, I managed to 'hack' it into life, however not ideal. Had to just match the driver version inside the container with the host, and then the ffmpeg inside the container just worked as expected with hardware acceleration. ? I think the release container needs more thought into how it manages its driver, should it reference the host only?, or at the very least attempt to match the host?.... needs investigation, because I can see things getting out of sync quite quickly with the drivers across container and host. |
|
The driver is pulled at build time, there is no easy way to automatically match the host driver. We should be pulling the most up to date driver for debian 11, odds are other OS have newer driver available and that is causing the issue. |
|
To be clear, just explaining how things are, I agree it is not a long term solution to work the way it does today. |
|
How did you go about matching the driver version inside the container? |
|
For now I have it configured to install 'intel-media-va-driver-non-free' of the latest version from Debian's 'testing' repo on every start. Not a big deal as it tends to stay running for around a month before restarting when I tend to upgrades/updates on the host and other containers. Not ideal indeed, alas have no other idea how to mitigate this issue. It is reasonable to assume the host driver could become out of sync on host updates/upgrades, hopefully some genius will come up with a solution to cover this stressful scenario as part of the release container. |
|
I have this exact same issue on intel gen12 CPU |
|
Seeing the same issue on 11th gen Intel. Tried a few different hwaccel args without any luck before i found this thread. rolling back to RC2 works, and disabling hwaccel also works. |
There have been no changes since RC2 concerning ffmpeg or any of its dependencies, not sure how that would be the case. |
Apologies, it was RC1. RC2 gave me green screens for the video feeds, so I rolled back to wait for a final and then figure out any issues then. |
|
@gbolo as discussed above you can use my test image |
thanks, that image seems to work as expected. What changes were made? |
The Intel iHD driver was updated to 22.3 from 22.2 |
|
Thanks @NickM-27 - driver fix resolves hardware issues on newer Alder Lake NUCs. |
Same here with Alder Lake i5-12500T using
@loe what |
|
Please provide error logs of QSV |
|
i'm also seeing some odd issues after the system has been running for a few hours (i'd say between 6-12 hours) then some of the streams seem to stop and never recover. I get a "Bus Error" eventually and it seems to give up. camera that seems to be in the error this time (I've seen it for others also) has this config. catgenie1: # <------ Name the camera
ffmpeg:
hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
inputs:
- path: rtsp://user:pass@ip:554/axis-media/media.amp
roles:
- detect
- rtmp
objects:
track:
- person
- catNo clue what the logs are saying here. |
|
@madasus Python Bus Error is known and covered in our docs: https://docs.frigate.video/faqs#fatal-python-error-bus-error |
|
thanks - trying that now |
Sure ! Here it is _frigate_logs.txt. A full run, from init to finish. |
|
@WhistleMaster thanks, if you run |
|
@NickM-27 I have the following: |
|
@WhistleMaster So you're on the correct version. Your error logs for qsv are different than the original errors here ( |
|
@NickM-27 I'm using docker on a debian: |
|
So the driver on your host is quite outdated ( |
|
@NickM-27 You're right ! Just updated to 22.6.1 on the host (from debian sid repo) but I get the same error. Shoud I create a new issue ? |
|
Most likely yes |
My CPU usage is still higher than I remember on older systems; but I do also see usage on intel_gpu_top. |
|
I tested the upgrade to [0.12.0-beta1] and ffmpeg went back to crashing. Do you know if the changes from the testing-driver image were included in the 12.0-beta1 docker? |
yes they were included, if I had to guess the crashing is unrelated to the driver |
|
going into the debug and then trying to run the ffmpeg command for one of the cameras manually inside the container yields this ffmpeg -hide_banner -loglevel warning -c:v h264_qsv -user_agent FFmpeg Frigate/0.12.0-53d39a1 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i "XXX/cam/realmonitor?channel=1&subtype=0" -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/backyardandpool1-%Y%m%d%H%M%S.mp4 -r 5 -s {} pipe: [NULL @ 0x555cfc411840] Unable to find a suitable output format for 'Frigate/0.12.0-53d39a1' I also have a few older cameras that were using VAAPI (in config i added this line Here is the ffmpeg debug for these with a slightly different error ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -user_agent FFmpeg Frigate/0.12.0-53d39a1 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i "XXX/cam/realmonitor?channel=1&subtype=0" -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/attic1-%Y%m%d%H%M%S.mp4 -r 5 -s {} pipe: Option hwaccel (use HW accelerated decoding) cannot be applied to output url Frigate/0.12.0-53d39a1 -- you are trying to apply an input option to an output file or vice versa. Move this option before the file it belongs to. |
|
I'd suggest making your own issue answering all the questions, this is entirely unrelated to this issue |
|
also going to close this as it is fixed in the beta |
|
using frigate:0.12.0-beta2 instead of frigate:stable seems to work for me |
|
BUMP - read all thread. Using NUC12. What is it exactly I need to do? |
|
This is an old issue on an old version of frigate, so the solutions here aren't relevant to the current version. |
|
I have the latest version and the issue. Where should I go? |
|
Create a new issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Describe the problem you are having
Hardware acceleration works on the host, just not inside the container in the new release 0.11.0.
If I roll back to v10, all is fine.
I know the FFMPEG build has now changed, and this HWACCEL issue is a bad nightmare for many projects, ever since the thread bugs to intel's split of i965 vs iHD etc.
I've spent a few hours on this, just can't seem to get this working, so gone back to v10 for now.
The error I get when attempting hardware acceleration inside frigate container via the hwaccel_args of '-hwaccel auto':
[h264 @ 0x55c00073de80] Failed to end picture decode issue: 23 (internal decoding error). [h264 @ 0x55c00073de80] hardware accelerator failed to decode picture Error while decoding stream #0:0: Input/output errorThe error I get when attempting accel inside frigate container via the hwaccel_args of '-hwaccel_output_format qsv -c:v h264_qsv -hwaccel_device /dev/dri/renderD128':
[h264_qsv @ 0x55fe09463f40] Error during QSV decoding.: device failed (-17) Error while decoding stream #0:0: Input/output errorThe output of vainfo inside the container
The output of vainfo on the host
inte_gpu_top works fine and shows the card as active inside the container, and when running ffmpeg with accel on the host, the gpu_top tool shows the appropriate hardware acceleration activity.
It seems like the FFMPEG build included in the 0.11.0 release of Frigate doesn't want to use Intel's Media iHD driver for hardware acceleration, though this is just a guess...
Version
0.11.0
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
n/aOperating system
Other Linux
Install method
Docker Compose
Coral version
PCIe
Network connection
Wired
Camera make and model
n/a
Any other information that may be helpful
No response
The text was updated successfully, but these errors were encountered: