3D Point Cloud Depth Estimation using Python

[Jaswanth987]

---

Required Info
Camera Model	{ D457 }
Firmware Version	(5.16.0.1)
Operating System & Version	Windows 10
Platform	PC
SDK Version	{ 2.55.1 }
Language	Python
Segment	others

Issue Description

Hii,
I am able to estimate depth using RealSenseViewer, and I want to estimate depth using Python instead. Using opencv_pointcloud_viewer.py and pyglet_pointcloud_viewer.py, I can visualize a 3D point cloud, but I cannot find the depth. Is there any example or source that can help me find depth using a 3D point cloud in Python? I am able to find depth in 2D using Python, but not in 3D.
Thanks in Advance.

[MartyG-RealSense]

Hi @Jaswanth987 The Python pointcloud script at #4612 (comment) may meet your requirements. It generates a pointcloud, stores its values in a numpy array and then uses the points.get_vertices instruction to retrieve the 3D X, Y and Z (depth) coordinates and print them.

[Jaswanth987]

Using the code mentioned, we will be able to retrieve all points of the point cloud. However, what I need instead is to visualize the 3D point cloud, and by hovering over or clicking on any point in the 3D point cloud, I need to get the depth at that point in python.
Thanks in advance

[MartyG-RealSense]

That kind of graphical user interface (GUI) is not straightforward to implement in Python. It would likely require an interface such as Pyglet or Tkinter (see #10337 (comment) for Tkinter resources).

The pyglet_pointcloud_viewer example would seem to be the best foundation to use for your project, as it already uses the same pointcloud generation and vertices retrieval method as the script that I linked to earlier. It just currently lacks the code line that displays the depth value with a print instruction.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/pyglet_pointcloud_viewer.py#L456-L463

The other mechanism that would need to be added is to print the value of a particular pointcloud coordinate when clicked or hovered on. I do not have RealSense Pyglet code for that kind of mouse interface though, unfortunately. It could possibly be triggered with a Pyglet On Mouse Press Event, as described at the link below.

https://www.geeksforgeeks.org/pyglet-on-mouse-press-event/

The position of the mouse cursor on the screen can also be tracked with On Mouse Motion.

https://pyglet.readthedocs.io/en/latest/programming_guide/mouse.html

[Jaswanth987]

Thank you for providing these approaches. I will review and try them out.

[Jaswanth987]

Hi @MartyG-RealSense

Is the following procedure correct?

1. Use pyglet_pointcloud_viewer to visualize a 3D point cloud.
2. Implement functionality to determine which point was clicked in the point cloud.
3. Using the 2D coordinates (x, y) of the click, obtain the 3D coordinates with rs.rs2_deproject_pixel_to_point.
4. The z-coordinate of these 3D coordinates will represent the depth.

Thanks in Advance.

[MartyG-RealSense]

You could implement it that way, yes.

[Jaswanth987]

Yes, it's working, thank you for that.

I have one more question. I am noticing occlusions in the stereo depth stream and the 3D point cloud. I have observed that there is an option in the RealSense Viewer called "Perform Occlusion Invalidation". Even after enabling this option, I am still experiencing occlusions. What might be the possible reasons for this?

Thank you in advance.

In 1st image and 3rd image you can see occlusions or reflections are present and 2nd image is rgb image of that.

[MartyG-RealSense]

It is not clear that the black areas of the image are all due to occlusion. It is a general physics principle (not specific to RealSense) that dark grey or black absorbs light and so makes it more difficult for depth cameras to read depth information from such surfaces. The darker the color shade, the more light that is absorbed and so the less depth detail that the camera can obtain.

So the blackness on the rear of the screen and the diagonal surface on the other side of the image could be because their color is too dark for the camera to read depth information from (the screen's slight reflectivity would help light to bounce back to the camera instead of the surface absorbing all the light, hence its blue depth color).

The black area to the side of the bottle could be due to the depth sensing being confused by its high reflectivity. And the black area of the foreground nearest to the camera could be because the minimum depth sensing distance of the camera will be around 0.5 meters / 50 cms, so that front area could be too close to the camera to register.

[MartyG-RealSense]

Hi @Jaswanth987 Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.