[D435] How do I replace depth information in realsense after some processing for display through Pyglet?

[adityashrm21]

Issue Description

I am using a D435 RealSense camera with the pointcloud_viewer.py script to collect and visualize depth information for windows on a building. Now since the camera gives a value of 0 (or non-sensical) for depth in the case of glasses and reflective surfaces, I am getting the depth information using depth_frame.get_data(), performing some processing on this and I want to update the visualization in Pyglet to be able to use this new depth for each frame it captures. Given that speed is not an issue and visualizing one frame at a time works, what's the best way to replace this new depth? I tried replacing values inside the verts array but that doesn't seem to help and seems incorrect. Any input on this would be appreciated.

Thanks.

[MartyG-RealSense]

Hi @adityashrm21 If there is not excessive reflection / glare from the glass then you may be able to improve your results by aligning depth to color. An example of doing this for an application involving glass is the fish aquarium in the depth_under_water Python tutorial.

https://github.com/IntelRealSense/librealsense/blob/jupyter/notebooks/depth_under_water.ipynb

The negative effects of glare from reflections could also be significantly reduced by applying a physical filter called a linear polarizer over the camera lenses. Section 4.4 of Intel's white-paper document about optical filters discusses this subject in detail.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400#section-4-the-use-of-optical-filters

By using one or both of the above methods, you may be able to avoid the need to use Pyglet.

[MartyG-RealSense]

Hi @adityashrm21 Do you still require assistance with this case, please? Thanks!

[adityashrm21]

Thanks for the input @MartyG-RealSense . These are helpful. I would still like to know if there is a way for us to modify the depth data inside a depth frame without converting it to numpy format or a way to get back to the frame structure after modifying it in the numpy format? I think I am looking for an alternative for depth_frame.set_data() method, because this method does not exist. Is there any way to do that?

[adityashrm21]

I think my issue is related to this one: #5784 but there has been no update on that.

[MartyG-RealSense]

Others have explored the subject of converting frame data to Python numpy array and the reverse of that process (numpy array to frame data), and it is an ongoing area of study for RealSense users. The link below may provide you with some research leads.

https://support.intelrealsense.com/hc/en-us/community/posts/360047819294/comments/360011874874

In regard to modifying the depth frame: using a post-processing filter in pyrealsense2 would accomplish that.

https://support.intelrealsense.com/hc/en-us/community/posts/360039839514-Modify-data-of-pyrealsense2-frame

[MartyG-RealSense]

Hi @adityashrm21 Do you require further assistance with this case, please? Thanks

[MartyG-RealSense]

Case closed due to no further comments received.