-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Capture PIDs die silently (possibly due to out of disk space condition). #1232
Comments
I ended up building a health check script for my instance that just loops through each camera and computes an average pixel color. If any are showing a green screen, I run some notification scripts.
|
without knowing what exception handling capabilities might already be in place; maybe something like Tenacity (https://tenacity.readthedocs.io/en/latest/) would be useful to consider? |
Edit: I cleaned up my disk (hundreds of GB free now) and restarted frigate. Unfortunately my cameras did not come back up - still got a green screen. I restarted docker - same issue. I ended up having to restart the entire host to get my feeds back. I'm a little at a loss. |
My camera feeds died overnight and I'm nowhere close to a full disk. Restarting Frigate did not get the feeds back, had to reboot the machine again. I'm beginning to wonder if this is somehow a GPU hwaccel driver issue, since the issue even survived restarting the docker daemon. Next time this happens I'll try to go down this particular garden path. This theory is further supported by the fact that my recordings never went down (they don't get decoded before being written to disk, therefore not going through ffmpeg's gpu-accelerated decoding). I think the RTMP streams served by Frigate also kept working fine. Good news is my hacky healthcheck listed above totally worked and notified me the cameras were down. |
observing the same behavior where stale/defunct capture PIDs are reported in the debug panel. I'm curious if others might be able to recreate this behavior by simply consoling into a running Frigate container, issuing a 'kill {active capture PID}' and then observing any resulting UI behavior. |
Happened again, all cams went green, restarting Frigate didn't help. I disabled hwaccel and everything came back. So I think at least part of this is caused by something in the intel hwaccel stack.
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Happened contemporaneously to #1231
I ran out of disk space, and all of my capture processes seem to have silently died. The UI is still running, no errors are reported, but the video shows up as pure green.
If I look at the debug panel, capture PIDs are still reported but they don't correspond to actual processes on the system.
For example:
Capture process started for driveway: 51
however PID 51 no longer exists on my system.I'm not sure what the best course of action is here, but logging when capture processes die would probably be a good start, and updating the debug screen so it no longer shows dead PID.
I'm not sure if it makes sense to automatically restart dead capture processes. It would be great to expose the service health via the API so for instance users can add monitoring to be notified when their cameras go dead.
The text was updated successfully, but these errors were encountered: