Intel real sense d455 frames = pipeline.wait_for_frames() RuntimeError: Frame didn't arrived within 5000

[js0011003]

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D455
Firmware Version	(05.13.00.50)
Operating System & Version	Win 10
Platform	PC
SDK Version	2.50.0
Language	python
Segment	others

Issue ### Description

i have my d455 set up and it worked normally yesterday. But when i ready to demo it to others an error happened. i have same code and same USB port, which i am confused what going wrong. My goal is tracking ones hip distance using mediapipe and real sense camera. Any help will be much appreciated. Here is my python code.

import cv2
import mediapipe as mp
import pyrealsense2 as rs
import numpy as np
import matplotlib.pyplot as plt

mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
postLmsStyle = mp_drawing.DrawingSpec(color=(0, 0, 255), thickness=2)
postConStyle = mp_drawing.DrawingSpec(color=(0, 255, 0), thickness=3)
mp_holistic = mp.solutions.holistic

pipeline = rs.pipeline()
config = rs.config()
pipeline.start(config)

align_to = rs.stream.color
align = rs.align(align_to)


// For webcam input:
cap = cv2.VideoCapture(0)
with mp_holistic.Holistic(
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5) as holistic:
 x = 0
 while cap.isOpened():
   success, image = cap.read()
   if not success:
     print("Ignoring empty camera frame.")
     # If loading a video, use 'break' instead of 'continue'.
     continue

 # To improve performance, optionally mark the image as not writeable to
 # pass by reference.
 while True:
     frames = pipeline.wait_for_frames()
     aligned_frames = align.process(frames)
     depth_frame = aligned_frames.get_depth_frame()
     color_frame = aligned_frames.get_color_frame()

 # depth = frames.get_depth_frame()

 # image size is 640 x 480

     color_image = np.asanyarray(color_frame.get_data())

     image.flags.writeable = False
     image_height, image_width, _ = image.shape
     image = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
     results = holistic.process(image)

  #  Draw landmark annotation on the image.
     image.flags.writeable = True
     image = cv2.cvtColor(color_image, cv2.COLOR_RGB2BGR)

     mp_drawing.draw_landmarks(
         image,
         results.pose_landmarks,
         mp_holistic.POSE_CONNECTIONS,
         postLmsStyle,
         postConStyle)

     dist_l = depth_frame.get_distance(int(results.pose_landmarks.landmark[mp_holistic.PoseLandmark.LEFT_HIP].x * image_width), 
     int(results.pose_landmarks.landmark[mp_holistic.PoseLandmark.LEFT_HIP].y * image_height))
     dist_r = depth_frame.get_distance(int(results.pose_landmarks.landmark[mp_holistic.PoseLandmark.RIGHT_HIP].x * image_width), 
     int(results.pose_landmarks.landmark[mp_holistic.PoseLandmark.RIGHT_HIP].y * image_height))
     dist = (dist_l + dist_r) / 2
     print(
         f'right hip distance is'
         f'{dist_r}'
         f'left hip distance is'
         f'{dist_l}'
         f'total distance is'
         f'{dist}'
         f'meters'
     )

     x = x + 0.04
     y = dist
     plt.scatter(x, y)
     plt.title("real time distance")
     plt.xlabel("time")
     plt.ylabel("distance")
     plt.pause(0.04)

 # Flip the image horizontally for a selfie-view display.
     cv2.imshow('MediaPipe Holistic', cv2.flip(image, 1))
     if cv2.waitKey(1) == ord('q'):
         plt.show()
         plt.pause(10)
         break

cap.release()

[MartyG-RealSense]

Hi @js0011003 The Frame didn't arrived within 5000 error indicates that the program may not be receiving frames from the camera and so is 'timing out' after a 5 second wait.

Is your camera able to stream frames in the RealSense Viewer normally, please?

[js0011003]

Yes, my D455 camera work normally in RealSense Viewer.

[MartyG-RealSense]

Thank you for the confirmation about the Viewer. That indicates that it is a pyrealsense2 related software problem rather than a camera hardware problem.

I have not seen a previous case of mediapipe being used with a pyrealsense2 program, though as the code worked up until the previous day I assume that the script code is not at fault.

Edit: I found another Python example of using mediapipe at #9088 where the hand is being tracked instead of the hip. Is the script at that link able to run if you test it?

[MartyG-RealSense]

Hi @js0011003 Do you require further assistance with this case, please? Thanks!

[negis02]

Hi, i also have same error, my app should run non stop, but sometimes after one or sometimes after 3 days exit whit this error. After start again, app continue running ok until next error.
Its happen on D435 with last firmware, ubuntu.

[MartyG-RealSense]

Hi @negis02 RealSense 400 Series cameras can run indefinitely so long as they remain within recommended operating temperature tolerances. These are 35 degrees C for the operating temperature and 50 degrees C for the temperature of the outer casing of the camera.

If your application has been running for 3 days and the camera temperatures are below the recommended maximums then typically the problem will be more likely to be related to a glitch on the computer hardware / OS or the computer's USB ports / cable rather than with the camera hardware.

The most recent supported kernels are 5.4 if installing the SDK from Debian packages, and 5.8 or 5.11 if building the SDK from source code with CMake. The SDK can work with unsupported kernels but there may be unpredictable consequences in regard to stability.

[MartyG-RealSense]

Hi @js0011003 and @negis02 Do you require further assistance with this case, please? Thanks!

[negis02]

Thanks for info, I will try to implement temperature monitor of cameras in my app.
Few day ago I move both cameras on new big PC (i7-12700), Before cameras was running on two separated identical PC (BOXNUC8i3BEK1). I'm going to think, that issue was caused by intel NUC USB3 ports hardware glitch's. With new PC looks like OK.

[MartyG-RealSense]

Thanks very much @negis02 for the update!

[MartyG-RealSense]

Hi @negis02 Do you require further assistance with this case, please? Thanks!

[negis02]

Thanks, for now running ok. I thinks it was Intel NUC PC reasons. So no need further assistance .

[MartyG-RealSense]

Thanks very much @negis02 for the update! As you do not require further assistance, I will close this case. Thanks again!

[Hanson-Chan]

I get this same problem.The file of .bag could be play in RealSense Viewer ,but it can't be extracted frames in pyrealsense2.By the way,I found that there is about 1.5 second(30fram/s) loss in the begining of depth video when i put the bag file on RealSense Viewer,which maybe show the reason why i can't extract frames in pyrealsense2?I don't know

[MartyG-RealSense]

Hi @Hanson-Chan If you wrote a script in pyrealsense2 to access bag frames, did you define playback.set_real_time(False) like in the Python script at #6340 as this can make bag playback more stable. The set_real_time instruction is True by default.

[Hanson-Chan]

@MartyG-RealSense Yes，i have done this

[MartyG-RealSense]

You could check whether there is a problem with the initial frames in the bag by programming the script to skip past the initial frames.

Assuming that the bag was recorded at 30 frames per second, the code below should skip the first 2 seconds (60 frames) of the bag. If your script has a 'wait_for_frames' instruction then the 'for i in range' instruction should be placed on the line before.

for i in range(60):
pipe.wait_for_frames()

[Hanson-Chan]

thanks for advice, for recycle is not useful for me .but i set playback.seek(datetime.timedelta(seconds=1.5)) to skip frames and problem solved

[MartyG-RealSense]

Thanks so much @Hanson-Chan for sharing your solution with the RealSense community!

[abdullahsaatci]

I have same problem too :( . I always have this error message when I use -- frames = pipeline.wait_for_frames():
Traceback (most recent call last):
File "3.py", line 19, in
ret, depth_frame, color_frame = dc.get_frame()
File "/home/abdullah/yazilim/realsense_depth.py", line 25, in get_frame
frames = self.pipeline.wait_for_frames()
RuntimeError: Frame didn't arrive within 5000

I can visualize my RealSense 435i camera on realsense-viewer and since I integrated it with ROS I can visualize it with rviz. Can you please help me to fix this issue or recommend me another way to use RealSense camera to use it with opencv-python. I need your help as soon as possible please.

[MartyG-RealSense]

Hi @abdullahsaatci Can you try the simple Python example program at the link below, please?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-tutorial-1-depth.py

If that works normally then it would suggest that there is a problem in your script. If you also experience Frame didn't arrived within 5000 with this example script too then it would indicate that your code is likely okay and the problem is elsewhere.

[abdullahsaatci]

Hi @abdullahsaatci Can you try the simple Python example program at the link below, please?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-tutorial-1-depth.py

If that works normally then it would suggest that there is a problem in your script. If you also experience Frame didn't arrived within 5000 with this example script too then it would indicate that your code is likely okay and the problem is elsewhere.

Hello @MartyG-RealSense ;
Thank you for replying when I run the code you suggested I receive this kind of output from terminal:

.nh
. ..
.nBn :hn:
:hnnnhh XW
:nn:nn:. :
nBn.nB. :nn. .
nXBnhBn::h: :XX:h:.
nBWXBBBXWn .::..nB..:
:BXXXWXXWXn:. . hh
.hWWhWWWXXn:
nXW: nXXBB:hhB:. n nh
:BXXh hWWX .BWWh .BWh h
.hXWBn nWB. nWWB. nn .::.
:XWhh nhXhn .: . :hWBWh.:hh :

                                                         :BB
                                                      .nn ::
                                                   .nXn.nBBh
                                                 :BBBXWWhBh:
                                              :BBXXXh:::nh. 
                                           .hXh:Xh.   ::    
                                         nBhhhXhB:  .:n  ...
                                      nBWXhBXn.    nn   : :.
                                   :BWXXWXWWh.         .. nn
                                .hXhhWXXBhn.                
                              nBWB.. .XWWXhBBX:.B.          
                           :BWXhh:.   hWWXhWWWWn:h  B.n:    
                        :hWWBnn:      :Xn  .nWWXhBnn.hh:n   
                       nXWWW                 nBWXW: nhn:    
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                          hn
                                                      .   .:
                                                   .nBn.:nn 
                                                 :BB:nBXn   
                                              :hhhnhhn   n:.
                                           .hXh.BB.   nBh   
                                         nXXBBXXX: . hWWXh. 
                                      :BXWXWh..:. .n .  ::::
                                   :BXWXXXXB.    :        :n
                                .hWXBWWWWX:             nn  
                              nXWXn. hXBWXnnhhn   :. .:h.   
                           :BWBBn:.  :hXWB.hWWX. n  n..     
                        :hWWX          nB  .:WWh. .:     .  
                       :XWBWhnn      .n:nh   .hWWWX .B.     
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                          hn
                                                      .n    
                                                   .nBh:nBh.
                                                 nBhnnBBn   
                                              :BBBXXB:   : .
                                           .hX::XWh.    hh: 
                                         nXBnhBB    :hBhXBB.
                                      :BWXWWn.:.   :B :.  .n
                                   :BWXXWWWB:..  :     . .h.
                                .hWWXWXWXBn     :.          
                              nXWX:  :XWWhnXXn. .Xh:        
                           :BWBB.    .BWXXBXhXh.    :.      
                        :hXXB ..      hWBn..:XWXn.h:.. .:h: 
                       :XWXXh                :nWBWh.hh:.:

But my main problem is visualizing colorized displaying not depth displaying, let me give you the details my problem please:

I can already run this script:

import pyrealsense2 as rs
import numpy as np
import cv2

Initialize the pipeline

pipeline = rs.pipeline()
config = rs.config()

config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

Start streaming

pipeline.start(config)

try:
while True:
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()

    if not depth_frame:
        continue
    
    # Convert the depth frame to a numpy array
    depth_image = np.asanyarray(depth_frame.get_data())
    
    # Display the depth image
    cv2.imshow('Depth Image', depth_image)
    
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

finally:
pipeline.stop()
cv2.destroyAllWindows()

but unfortunately whenever I add this line to script
"config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)"

I receive the runtime error message did not arrive in 5000 which I mentioned earlier.

I can visualize both depth images and colorized images on RealSense-viewer and rviz.

My RealSense depth camera is -- 435i

I have set uo the sdk by fallowing these steps if you wonder -- https://eu.mouser.com/applications/getting-started-with-realsense-d455/

[MartyG-RealSense]

What happens if you remove config from the brackets of the pipe start instruction so that the 'config' lines of the script are ignored?

pipeline.start()

The program should then automatically apply the default stream configuration of the D435i camera - 848x480 depth at 30 FPS and 1280x720 RGB color at 30 FPS. If the program runs correctly then it would suggest that the program has a problem with the config lines for some reason, even though their code is correct.

[abdullahsaatci]

What happens if you remove config from the brackets of the pipe start instruction so that the 'config' lines of the script are ignored?

pipeline.start()

The program should then automatically apply the default stream configuration of the D435i camera - 848x480 depth at 30 FPS and 1280x720 RGB color at 30 FPS. If the program runs correctly then it would suggest that the program has a problem with the config lines for some reason, even though their code is correct.

nothing changes same error as before :/

[MartyG-RealSense]

Let's try using RGB from the left infrared sensor instead of RGB from the RGB sensor, as the D455 camera model supports 'RGB from left infrared' mode.

config.enable_stream(rs.stream.infrared, 640, 480, rs.format.bgr8, 30)

[abdullahsaatci]

Let's try using RGB from the left infrared sensor instead of RGB from the RGB sensor, as the D455 camera model supports 'RGB from left infrared' mode.

config.enable_stream(rs.stream.infrared, 640, 480, rs.format.bgr8, 30)

I think no way to fix the issue

[MartyG-RealSense]

Do you still get RuntimeError: Frame didn't arrived within 5000 if you insert the Python camera reset code below at the line before your pipe start line?

ctx = rs.context()
devices = ctx.query_devices()
for dev in devices:
dev.hardware_reset()

[abdullahsaatci]

ctx = rs.context()
devices = ctx.query_devices()
for dev in devices:
dev.hardware_reset()

now it sometimes gives this error :

abdullah@abdullah:~/yazilim$ python3 main.py
Traceback (most recent call last):
File "main.py", line 19, in
pipeline.start(config)
RuntimeError: map_device_descriptor Cannot open '/dev/video0 Last Error: No such file or directory

and sometimes:

abdullah@abdullah:~/yazilim$ python3 main.py
Traceback (most recent call last):
File "main.py", line 19, in
pipeline.start(config)
RuntimeError: Couldn't resolve requests

[abdullahsaatci]

I receive this error too:
abdullah@abdullah:~/yazilim$ python3 main.py
Traceback (most recent call last):
File "main.py", line 24, in
frames = pipeline.wait_for_frames()
RuntimeError: Frame didn't arrive within 5000

[MartyG-RealSense]

There was a case at #10570 (comment) where the map_device_descriptor error was being caused by including the hardware_reset() instruction in the script.

RuntimeError: Couldn't resolve requests is caused because the camera is unable to provide the particular 'config' stream configuration that was requested. However, your depth config line was working and you only had a problem when adding a color config line.

config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

We established that the depth stream is able to be accessed because you were able to run the python-tutorial-depth-1.py.

Can you confirm whether or not you were able to stream color from infrared instead of from the RGB instruction using the config line at #10523 (comment)

[abdullahsaatci]

There was a case at #10570 (comment) where the map_device_descriptor error was being caused by including the hardware_reset() instruction in the script.

RuntimeError: Couldn't resolve requests is caused because the camera is unable to provide the particular 'config' stream configuration that was requested. However, your depth config line was working and you only had a problem when adding a color config line.

config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

We established that the depth stream is able to be accessed because you were able to run the python-tutorial-depth-1.py.

Can you confirm whether or not you were able to stream color from infrared instead of from the RGB instruction using the config line at #10523 (comment)

when I run this code you mentioned

import pyrealsense2 as rs
import cv2
import numpy as np

try:
pipeline = rs.pipeline()

config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.infrared, 640, 480, rs.format.bgr8, 30)
pipeline.start(config)

while True:
    frames = pipeline.wait_for_frames()
    depth = frames.get_depth_frame()
    if not depth:
        continue

    depth_image = np.asanyarray(depth.get_data())
    depth_value = depth.get_distance(320,280)
    print(depth_value)
    cv2.imshow('Depth Image', depth_image)
    
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

finally:
pipeline.stop()
cv2.destroyAllWindows()

I receive this error

Traceback (most recent call last):
File "learn_depth.py", line 27, in
pipeline.start(config)
RuntimeError: Couldn't resolve requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "learn_depth.py", line 53, in
pipeline.stop()
RuntimeError: stop() cannot be called before start()

But when I remove this line of script
config.enable_stream(rs.stream.infrared, 640, 480, rs.format.bgr8, 30)
everything works perfect
thanks in advance

[MartyG-RealSense]

What happens if you enable the color stream without setting any configuration values for it?

config.enable_stream(rs.stream.color)

[abdullahsaatci]

config.enable_stream(rs.stream.color)

I receive this error again

abdullah@abdullah:~/yazilim$ python3 learn_depth.py
Traceback (most recent call last):
File "learn_depth.py", line 34, in
frames = pipeline.wait_for_frames()
RuntimeError: Frame didn't arrive within 5000

[MartyG-RealSense]

I wonder if the lines below are breaking the script when the color stream is included.

if not depth:
 continue

It is basically saying to prevent the script from continuing to work if there is a stream type enabled that is not the depth stream.