Aligning and filling array pyrealsense2

[30Ronaldo]

---

Required Info
Camera Model	D435i
Operating System & Version	Win10
Platform	PC
SDK Version	latest
Language	python

Issue Description

I wrote a python script to record color and depth images from Realsense D435i. To avoid latency I am saving the color images as video. The raw depth frames should be stored in arrays, to extract the 3D coordinates of each pixel after recording. Anyway before saving them I am additionally aligning the depth frames to the color ones.
Running this script causes after exactly 15 frames following error:

          alignedframes = self.align.process(frames)
          RuntimeError: Error occured during execution of the processing block! See the log for more 
          info

Running it without the aligning function works. Running it without the line, where I store the frames in the array, works. Running it with alignment of COLOR to DEPTH works.
It seems, that the alignment function (Depth -> Color) and the array storing of the resulting depth frames collide with each other curiously at the 16th frame.

I also tried to save the 3D coordinates of each pixel in preallocated numpy array within the recording loop. This method causes slightly latencies and the defined sample rate of 30Hz is no longer given.
Here the important pieces of the script:

self.pipeline = rs.pipeline()
self.config = rs.config()

self.config.enable_stream(rs.stream.depth, w, h, rs.format.z16, fr)
self.config.enable_stream(rs.stream.color, w, h, rs.format.bgr8, fr)

self.pc = rs.pointcloud()
align_to = rs.stream.color
self.align = rs.align(align_to)

self.Depth = [None]*1000

def record(self):
    while self.runFlag:
        frames = self.pipeline.wait_for_frames()
        if frames.size() < 2:
            continue
        alignedframes = self.align.process(frames)
        depth_frame = alignedframes.get_depth_frame()
        color_frame = alignedframes.get_color_frame()
        if not depth_frame or not color_frame:
            continue

        self.timestamp.append(datetime.now().timestamp() * 1000)
        self.Depth[self.counter] = depth_frame
        color_image = np.asanyarray(color_frame.get_data())

        self.vw.write(color_image)
        self.counter += 1

[MartyG-RealSense]

Hi @30Ronaldo How does the script behave if you remove these lines, please:

if frames.size() < 2:
continue

[30Ronaldo]

Same result, I added these lines after recognizing this error to check if this may help.

[MartyG-RealSense]

Apologies for the delay in responding further. I was carefully considering how to proceed.

The code that you used to obtain the color and depth is consistent with examples of alignment for 2D frames. I wonder if it requires further steps for aligning a 3D point cloud though.

Edit: here is someone else's approach to 3D point cloud alignment in Python:

#2769 (comment)

[30Ronaldo]

Thanks for your response.

I don't know how this could solve my problem. I am not losing information about the pointcloud by aligning the belonging depth frame.

Nevertheless I would like to know why this error is raising while aligning and storing depth frames.

[MartyG-RealSense]

The link below has an example of aligning depth and color and making use of a numpty array.

#3658

If this does not help then I will consult with a support manager about your case.

[30Ronaldo]

As mentioned, I would like to save the depth frames in an array for better performance.
Additionally I can add, that saving the color frames in an array works, but again after 15 frames something goes wrong and it stops recording. Thereafter the frames don't change.

[30Ronaldo]

As mentioned, I would like to save the depth frames in an array for better performance.
Additionally I can add, that saving the color frames in an array works, but again after 15 frames something goes wrong and it stops recording. Thereafter the frames don't change.

[MartyG-RealSense]

I located a case where another user could get no more than 15 frames. Please check the advice given in that case to see if it is relevant to your own situation.

#946 (comment)

[30Ronaldo]

I changed the line in source.cpp and rebuild the project but it still stops after 16 frames.
But good to know the reason for this behaviour. Maybe I should go back to the option saving it all in .bag files.

[MartyG-RealSense]

This case will be closed after 7 days from the time of writing this message if there are no further comments. Thanks!