rs.points.get_vertrices() returns wrong type

[FloWsnr]

---

Required Info
Camera Model	D435i
Firmware Version	RealSense Viewer 2.50.0.3785
Operating System & Version	Win 10
Platform	PC
SDK Version	2.
Language	python
Segment	Robot

Issue Description

I am currently trying to create an open3D pointcloud via the realsense SDK. However, I receive this:

RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)

The code below shows the problem: vtx = np.asanyarray(points.get_vertices()) does not return a normal numpy array. Therefore, the o3d utility o3d.utility.Vector3dVector(vtx) fails. I already tested the o3d utility with normal numpy arrays. They work fine.
Inspecting via the debugger shows, that vtx is an array of dtype([('f0', '<f4'), ('f1', '<f4'), ('f2', '<f4')])

What am I doing wrong here? Thanks for the help!

import pyrealsense2 as rs
import numpy as np
import open3d as o3d

pipeline = rs.pipeline()
config = rs.config()

cfg = pipeline.start(config)
dev = cfg.get_device()

pcd = o3d.geometry.PointCloud()
pc = rs.pointcloud()
points = rs.points()
try:
    while True:

        frames = pipeline.wait_for_frames()
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()
        if not depth_frame or not color_frame:
            continue

        pc.map_to(color_frame)
        points = pc.calculate(depth_frame)
        vtx = np.asanyarray(points.get_vertices())

        pcd.points = o3d.utility.Vector3dVector(vtx)

finally:
    pipeline.stop()

[MartyG-RealSense]

Hi @FloWsnr A RealSense Open3D user at isl-org/Open3D#473 shares their Python pointcloud scripting, using what seems to be a similar approach to your own. In the script under the What I tried heading, they are using Vector3DVector(vtx.tolist()) instead of just vtx. Does their script run correctly if you test it, please?

[FloWsnr]

Hey MartyG,

Thanks for the fast help. Indeed, adding tolist() works! Maybe for other people trying something similar.
You can get the same result by using this:

depth_image = np.asanyarray(depth_frame.get_data())
depth_o3d_image = o3d.geometry.Image(depth_image)

pinhole_camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(intr.width, intr.height, intr.fx, intr.fy, intr.ppx, intr.ppy)
pcd = o3d.geometry.PointCloud().create_from_depth_image(depth_o3d_image, pinhole_camera_intrinsic, depth_scale = 1000)

Disadvantage is the need for the camera intrinsics, which you can get by using:

pipeline = rs.pipeline()
config = rs.config()
cfg = pipeline.start(config)
profile = cfg.get_stream(rs.stream.depth) # Fetch stream profile for depth stream
intr = profile.as_video_stream_profile().get_intrinsics() # Downcast to video_stream_profile and fetch intrinsics

[MartyG-RealSense]

It's great to hear that adding tolist() worked for you, @FloWsnr - thanks very much for the update and for sharing your solution with the RealSense community!