D435i: transform infrared pointcloud to rgb pointcloud

[badob]

Required Info
Camera Model	D400 (D435i)
Firmware Version	-
Operating System & Version	Linux (Ubuntu 20)
Kernel Version (Linux Only)	-
Platform	PC
SDK Version	-
Language	Python / C++
Segment	Robot
Framework / Library	ROS / Open3D / OpenCv

Issue Description

Hi, i want to transform my infra1 pointcloud to my rgb pointcloud. Is there a way to transform these?

I created my rgb pointcloud from the color and aligned_depth_to_color frame with Open3D and ROS. For my infra1 pointcloud i used infra1 and depth frame. Because of the field of view, one point cloud is now larger than the other. I just need the size of the rgb pointcloud.

How can I crop the pointcloud to fit the other and then transform a point from rgb to infra1 pointcloud?

Or do I have to match the rgb and infra1 frame first and then create the pointclouds?

Can i use some of these (Rostopic):

/camera/extrinsics/depth_to_color

header: 
  seq: 0
  stamp: 
    secs: 0
    nsecs:         0
  frame_id: "depth_to_color_extrinsics"
rotation: [0.9998142719268799, -0.01877661421895027, -0.004348050802946091, 0.018780462443828583, 0.999823272228241, 0.0008461546967737377, 0.00433139456436038, -0.0009276559576392174, 0.9999901652336121]
translation: [0.014845957048237324, -0.00017637049313634634, 0.00021370494505390525]

/camera/aligned_depth_to_color/camera_info

header: 
  seq: 8232
  stamp: 
    secs: 1667313088
    nsecs: 559242010
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [606.599853515625, 0.0, 328.7848205566406, 0.0, 605.3248291015625, 252.768310546875, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [606.599853515625, 0.0, 328.7848205566406, 0.0, 0.0, 605.3248291015625, 252.768310546875, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

[MartyG-RealSense]

Hi @badob The problem seems to be that when depth is aligned to color on the RGB pointcloud, the FOV size is changing because the depth FOV is being resized to the D435i's smaller color FOV.

On the other point cloud, depth and infra1 originate from the same sensor (left infrared) and should therefore have the same FOV size when aligned. So there would be FOV differences between the two clouds.

For the RGB cloud, a solution may be to use color to depth alignment instead of depth to color, so that the color FOV is resized to match the larger depth FOV. However, the RealSense ROS wrapper does not have a color to depth alignment function (only depth to color). So whilst color to depth alignment might be possible with Open3D scripting, it would not be practical to achieve when using ROS topics.

If you wish to explore the cropping approach, there is an example of ROI cropping with Open3D at #6267 (comment)

To transform between two pointclouds of different sizes with Open3D, ICP registration may be a viable option, as described at isl-org/Open3D#5082

[MartyG-RealSense]

Hi @badob Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.