Potentially damaged during conversion to uint16 depth images using Intel RealSense D435??

[monajalal]

Are these depth maps damaged and if so, how can I fix them? I need them to be 16bit and I need to multiply them by 1000 as author of BundleSDF mentioned.

corresponding rgb image is:

from pathlib import Path

from rosbags.highlevel import AnyReader
import numpy as np
from PIL import Image
from datetime import datetime
from matplotlib import image
import cv2

import matplotlib.pyplot as plt


# create reader instance and open for reading

with AnyReader([Path('/home/mona/rosbag2_2023_11_06-15_44_24')]) as reader:


    connections = [x for x in reader.connections if x.topic == '/camera/camera/aligned_depth_to_color/image_raw']

    for connection, timestamp, rawdata in reader.messages(connections=connections):
        msg = reader.deserialize(rawdata, connection.msgtype)

        timestamp_dt = datetime.fromtimestamp(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9)
        timestamp_str = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S.%f")

        # Convert the timestamp to nanoseconds
        timestamp_ns = msg.header.stamp.sec * 1e9 + msg.header.stamp.nanosec

        # Convert the timestamp to the desired numeric representation
        numeric_timestamp = int(timestamp_ns / 1e-9)

        

        image_data = msg.data.reshape(480, 640,-1)*1000#.astype(np.uint16)
        
        # Take only the first channel (grayscale)
        grayscale_image = image_data[:, :, 0]

        fig, ax = plt.subplots()

        ax.axis('off')

  
        # Save the depth image as a PNG file
        depth_image_name = 'depth/' + str(numeric_timestamp)[:20] + '.png'
        cv2.imwrite(depth_image_name, grayscale_image.astype(np.uint16))


        
 
        
plt.close('all')

currently, this yields with an example depth image with an identifier like this:

$ identify -verbose 16993034780804345302.png
Image:
  Filename: 16993034780804345302.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: DirectClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: Gray
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    gray: 16-bit
  Channel statistics:
    Pixels: 307200
    Gray:
      min: 0  (0)
      max: 65464 (0.998917)
      mean: 28581.5 (0.436126)
      standard deviation: 20823.3 (0.317743)
      kurtosis: -1.33259
      skewness: 0.0241377
      entropy: 0.898635
  Colors: 256

Here's an example of depth images from BundleSDF author:

$ identify -verbose 1668813100171987935.png
Image:
  Filename: 1668813100171987935.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: DirectClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: Gray
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    gray: 16-bit
  Channel statistics:
    Pixels: 307200
    Gray:
      min: 0  (0)
      max: 917 (0.0139925)
      mean: 322.928 (0.00492757)
      standard deviation: 337.252 (0.00514614)
      kurtosis: -1.83116
      skewness: 0.164199
      entropy: 0.58544
  Colors: 414

NVlabs/BundleSDF#82 (comment)

NVlabs/BundleSDF#82 (comment)

NVlabs/BundleSDF#74 (comment)

[MartyG-RealSense]

Hi @monajalal Another RealSense user who was trying to export a 16-bit depth image had a similarly incorrect image at IntelRealSense/librealsense#7081 (comment)

They then tried using OpenCV code at IntelRealSense/librealsense#7081 (comment) to perform the conversion of the RealSense image to 16-bit mat.

Mat image16(Size(w, h), CV_16UC1, (void*)depth.get_data(), Mat::AUTO_STEP);

[monajalal]

Can you please show the equivalent of the C++ code you wrote above embedded in the Python code below? I did some searching didn't find the equivalent of CV_16UC1 in python cv2. Also, the author of BundleSDF suggested to multiply the depth values by 1000 and also save them as np.uni16 using astype. Any chance you could share how to change the code below?

from pathlib import Path
from rosbags.highlevel import AnyReader
import numpy as np
from PIL import Image
from datetime import datetime
from matplotlib import image
import cv2
import matplotlib.pyplot as plt


# create reader instance and open for reading
with AnyReader([Path('/home/mona/rosbag2_2023_11_06-15_44_24')]) as reader:

    # connections = [x for x in reader.connections if x.topic == '/camera/camera/color/image_raw']
    connections = [x for x in reader.connections if x.topic == '/camera/camera/aligned_depth_to_color/image_raw']

    for connection, timestamp, rawdata in reader.messages(connections=connections):
        msg = reader.deserialize(rawdata, connection.msgtype)
        timestamp_dt = datetime.fromtimestamp(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9)
        timestamp_str = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S.%f")

        # Convert the timestamp to nanoseconds
        timestamp_ns = msg.header.stamp.sec * 1e9 + msg.header.stamp.nanosec
        # Convert the timestamp to the desired numeric representation
        numeric_timestamp = int(timestamp_ns / 1e-9)
        # Reshape the image data to match the desired size
        image_data = msg.data.reshape(480, 640,-1)*1000
        
        # Take only the first channel (grayscale)
        grayscale_image = image_data[:, :, 0]
        depth_image_name = 'depth/' + str(numeric_timestamp)[:20] + '.png'
        cv2.imwrite(depth_image_name, grayscale_image.astype(np.uint16))
       

Also, could you please let me know what is the English term for the problem in the depth image below? What has caused it? I don't see a problem when I do depth align raw depth image in rviz2.

I am curious what line of code has caused this problem?

[MartyG-RealSense]

I was not able to find a Python equivalent of the C++ code either, unfortunately. I believe that this is because Python handles conversion of RealSense data to cv:: differently by accessing data stored in a numpy array and then rendering it as a mat image.

The script at IntelRealSense/librealsense#3658 may be a useful Python reference for accessing depth data stored in a numpy array and saving it to a PNG file.

I do not have a suggestion to offer about adapting the Python script above either and do not know about BundleSDF. I have also not previously seen others using the approach of multiplying by 1000 to obtain a 16-bit CV image. I sincerely apologize.

[monajalal]

@MartyG-RealSense thanks for sharing the link.

When I use this script https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

I get so much noise in my capture (light gray colors and such on the blue cup and the object is quite close to the D435 camera). I don't see this problem when I visualize my depth aligned capture by ros inside rviz2.

(capture) mona@ada:~/depth_aligned_capture_realsense$ python align-depth2color.py 
Depth Scale is:  0.0010000000474974513

and here is what I see if I do depth aligned capture in rviz2 using ros2 humble

Further, if it may deem helpful, I documented my question with all details in one post here https://stackoverflow.com/questions/77439308/fixing-the-repetitive-wave-structure-in-depth-map-when-writing-to-disk

[monajalal]

Also @MartyG-RealSense I saw your response here:
#1485 (comment)
could you please let me know if any of these lines are wrong in achieving the purpose of saving depth map from RealSense as uint16? I am not fully sure if I should use them

-1 here image_data = msg.data.reshape(h, w,-1)*1000

and

taking only the first channel as in grayscale_image = image_data[:, :, 0]

I wonder if any of these two lines is causing it?

Thanks for your feedback.

[MartyG-RealSense]

In close range situations where there are gray areas overlaid then the image may improve if the depth scale is changed from the default 0.001 to 0.0001. Python code for setting the depth unit value can be found at IntelRealSense/librealsense#10976 (comment)

In regard to the second question, I do not know but IntelRealSense/librealsense#5414 may be a helpful Python reference in regard to converting 16-bit data and saving it as a PNG.

# Convert 16bit data
detph_gray_16bit = np.array(depth_gray_image, dtype=np.uint16)
detph_gray_16bit *= 256

[monajalal]

I have an update that I fixed my problem this way (our depth histograms look weirdly different)

from pathlib import Path
from rosbags.highlevel import AnyReader
import numpy as np
from PIL import Image
from datetime import datetime
from matplotlib import image
import cv2
import matplotlib.pyplot as plt

with AnyReader([Path('/home/mona/rosbag2_2023_11_06-15_44_24')]) as reader:
    connections = [x for x in reader.connections if x.topic == '/camera/camera/aligned_depth_to_color/image_raw']
    for connection, timestamp, rawdata in reader.messages(connections=connections):
        msg = reader.deserialize(rawdata, connection.msgtype)
        timestamp_dt = datetime.fromtimestamp(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9)
        timestamp_str = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S.%f")
        timestamp_ns = msg.header.stamp.sec * 1e9 + msg.header.stamp.nanosec
        numeric_timestamp = int(timestamp_ns / 1e-9)
        w, h = msg.width, msg.height
     
        image_data = msg.data.reshape(h, w,-1)

        
        depth = cv2.inRange(image_data, 0.1, 2.0)
        depth_scaled = (depth*1000).astype(np.uint16)
        depth_image_name = 'depth/' + str(numeric_timestamp)[:20] + '.png'
        print('image_data.shape: ', image_data.shape)
        depth_data = np.array(image_data, dtype=np.uint16)
        cv2.imwrite(depth_image_name, depth_scaled)
 

specifically, this was the main change:

        depth = cv2.inRange(image_data, 0.1, 2.0)
        depth_scaled = (depth*1000).astype(np.uint16)

NVlabs/BundleSDF#115 --> please let me know about your thoughts on why the depth histograms are wildly different?

[MartyG-RealSense]

@monajalal It's great to hear that you achieved a solution. Thanks very much for sharing the details of it!

[monajalal]

@MartyG-RealSense thanks a lot for your openness in the community and providing excellent service to the customers.

[MartyG-RealSense]

You are very welcome. Thanks to you too for your kind words :)