There is an error when generating point clouds with "rs2::pointcloud::calculate()".

[Shannon-Law]

When I use the codes from https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl/rs-pcl.cpp, and compile them in a Jetson board, to generate point clouds, all the point coordinates of "points = pc.calculate(depth);" are zeros, while the depth maps are normal. What's more, the camera I use is RealSense D435, and the SDK version is 2.50.0. Why is that and how to solve it?

[MartyG-RealSense]

Hi @Shannon-Law Does you RealSense camera model have an RGB sensor? If it does, do you experience similar problems with the rs-pcl-color example?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl-color/rs-pcl-color.cpp

[Shannon-Law]

@MartyG-RealSense Yes, I have tried. And the same error like it above, exists.

[MartyG-RealSense]

Can you confirm please that you installed the RealSense PCL wrapper using the installation instructions at the link below, please?

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/pcl

[Shannon-Law]

@MartyG-RealSense Yes, I confirmed. However, the source of the problem resides in the fact that the outputs of the function "rs2::pointcloud::calculate()", namely, the values of "points = pc.calculate(depth)", are zeros. It has nothing to do with PCL.

The codes below from
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl/rs-pcl.cpp
or
https://github.com/IntelRealSense/librealsense/blob/master/examples/pointcloud/rs-pointcloud.cpp
are used to generate point clouds.

rs2::pointcloud pc;
rs2::points points;
rs2::pipeline pipe;
pipe.start();
while(true){
auto frames = pipe.wait_for_frames();
auto depth = frames.get_depth_frame();
points = pc.calculate(depth);
//The values of "points = pc.calculate(depth)", are zeros.
}

[MartyG-RealSense]

Your simple code listing above looks okay. Do you receive any depth values with the script below?

rs2::pointcloud pc;
rs2::points points;
rs2::pipeline pipe;
pipe.start();
while(true){
auto frames = pipe.wait_for_frames();
auto depth = frames.get_depth_frame();
auto color = frames.get_color_frame();
pc.map_to(color);
points = pc.calculate(depth);
}

If your depth values are still zeroes, would it be possible to provide an RGB image of the scene that the camera is observing to see whether there are aspects of the scene that may make it difficult for the camera to analyze it for depth information, please?

[Shannon-Law]

In different scenes, the depth values are normal, but all the coordinates of the generated point clouds are zeros.

[MartyG-RealSense]

I researched this problem extensively but was unable to find a potential cause for your particular problem, unfortunately.

Are you able to generate a pointcloud in the 3D mode of the RealSense Viewer tool (realsense-viewer) please?

You can generate a depth-only pointcloud by enabling the depth stream in the Viewer and going to 3D mode using the '3D' option in the top corner of the Viewer window, or a color-textured pointcloud by first enabling depth and then enabling RGB secondly.

[Shannon-Law]

@MartyG-RealSense @RealSenseCustomerSupport
In RealSense Viewer, the depth maps and the point clouds are normal. But, the generated point cloud with the function "rs2::pointcloud::calculate()", are abnormal, when I use the codes above.

[MartyG-RealSense]

The RealSense Viewer applies a range of post-processing and depth colorization presets by default, so a depth image in the Viewer may not resemble one generated by a self-created script, as the preset settings applied by default in the Viewer have to be deliberately programmed into a script manually to replicate the appearance of a Viewer image.

Can you describe how the point cloud provided by rs2::pointcloud::calculate() looks abnormal please? Thanks!

[Shannon-Law]

In the codes below, the outputs of the function "rs2::pointcloud::calculate()", namely, the values of "points = pc.calculate(depth)", are zeros. But, in RealSense Viewer, the depth maps and the point clouds are normal.

The codes below from
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl/rs-pcl.cpp
or
https://github.com/IntelRealSense/librealsense/blob/master/examples/pointcloud/rs-pointcloud.cpp
are used to generate point clouds.

rs2::pointcloud pc;
rs2::points points;
rs2::pipeline pipe;
pipe.start();
while(true){
auto frames = pipe.wait_for_frames();
auto depth = frames.get_depth_frame();
points = pc.calculate(depth);
//The values of "points = pc.calculate(depth)", are zeros.
}

[MartyG-RealSense]

If the librealsense SDK was built with example programs included, could you check at the folder location usr/local/bin whether there is a pre-built version of the rs-pointcloud example that can be launched to see whether this pre-built version of rs-pontcloud.cpp has the same problem with zeroes?

[Shannon-Law]

At the folder location usr/local/bin of my Jetson system, there is not any pre-built version of the rs-pointcloud example.

[MartyG-RealSense]

Okay, thank you. If the SDK is built from source code with CMake then including the build flag DBUILD_GRAPHICAL_EXAMPLES=true in the CMake build instruction should include executable versions of the SDK's set of graphics-using example programs and tools in the build.

The problem that you are experiencing sounds similar to the one in #5728

[Shannon-Law]

Yes, this error is similar to the one in #5728. And I install the SDK through the way described in https://github.com/IntelRealSense/librealsense/blob/c94410a420b74e5fb6a414bd12215c05ddd82b69/doc/installation_jetson.md, instead of using source codes with CMake.

@MartyG-RealSense Thanks for your patience and reply!

[MartyG-RealSense]

You are very welcome!

In #5728 the problem was occurring for that user only when they ran the code on a virtual machine (VM). If they ran the code directly on a Jetson Nano instead of using a VM then it worked correctly.

Are you using a VM, please?

[Shannon-Law]

No, I'm not using a virtual machine. Instead, I run the codes above directly on a Jetson TX2 and a Jetson AGX Xavier.

[MartyG-RealSense]

Here's another C++ example to try:

#include <librealsense2/rs.hpp>
#include <iostream>
int main()
{
    rs2::pointcloud pc;
    rs2::points points;
    rs2::pipeline pipe;
    pipe.start();
    auto frames = pipe.wait_for_frames();
    auto depth = frames.get_depth_frame();
    points = pc.calculate(depth);
    auto vertices = points.get_vertices();
    auto tex_coords = points.get_texture_coordinates();
    long int i = 0;
    while (1)
    {
	i ++;
	std::cout<<i<<std::endl;
        frames = pipe.wait_for_frames();
        depth = frames.get_depth_frame();
        points = pc.calculate(depth);
        vertices = points.get_vertices();
        tex_coords = points.get_texture_coordinates();
    }
}

[Shannon-Law]

Thank you so much! I tested the codes above and printed the coordinates on my Jetson AGX Xavier. Same as before, all the values of "vertices" and "tex_coords" are zeros.

[MartyG-RealSense]

If pc.calculate is not working at all, let's next try getting the coordinates printed with rs2_deproject_pixel_to_point to confirm whether the problem is with pc.calculate in particular or all point cloud scripts.

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <librealsense2/rs_advanced_mode.hpp>
#include <librealsense2/rsutil.h>
#include "../example.hpp" // Include short list of convenience functions for rendering

#include <algorithm> // std::min, std::max
#include <string>
#include <iostream>

#include <time.h>
//#include <sys/time.h>
#include <stdlib.h>
#include <stdio.h>

using namespace rs2;
using namespace std;
// Helper functions
void register_glfw_callbacks(window& app, glfw_state& app_state);

int main(int argc, char * argv[]) try
{
 // Create a simple OpenGL window for rendering:
 window app(1280, 720, "RealSense Pointcloud Example");
 // Construct an object to manage view state
 glfw_state app_state;
 // register callbacks to allow manipulation of the pointcloud
 register_glfw_callbacks(app, app_state);

 pipeline pp;
 config cfg;
 float ResultVector[3];
 float InputPixelAsFloat[2];
 InputPixelAsFloat[0] = 416;
 InputPixelAsFloat[1] = 316;

 cfg.enable_stream(RS2_STREAM_DEPTH, 640, 480, RS2_FORMAT_Z16 , 30);
 cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_RGB8, 30);
 auto MyPipelineProfile = pp.start(cfg);

 rs400::advanced_mode MyDevice = MyPipelineProfile.get_device();
 rs2::depth_sensor DepthSensor = MyDevice.first<rs2::depth_sensor>();
 float DepthUnits = DepthSensor.get_option(RS2_OPTION_DEPTH_UNITS);

 // get & store depth intrinsics
 auto DepthStream = MyPipelineProfile.get_stream(RS2_STREAM_DEPTH).as<rs2::video_stream_profile>();
 rs2_intrinsics DepthIntrinsics = DepthStream.get_intrinsics();
 // convert DistanceToTarget from "Cm" to camera's depth scale

 for (int i = 0; i<10; i++)//to skip first few frames when device just initiated
  auto frames = pp.wait_for_frames();

 auto frames = pp.wait_for_frames();

 rs2::depth_frame depthFrame = frames.get_depth_frame();
 float distance = depthFrame.get_distance(416, 316);

 std::cout << " !!! depthunits = " << DepthUnits << " DistanceToTargetInDepthScale = " << distance << std::endl;

 // using librealsense's built in function for this

 rs2_deproject_pixel_to_point(ResultVector, &DepthIntrinsics, InputPixelAsFloat, distance);

 std::cout << "x = " << ResultVector[0] << ", y = " << ResultVector[1] << ", z = " << ResultVector[2] << std::endl;

 std::cout << "end" << std::endl;

 return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
 std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
 return EXIT_FAILURE;
}
catch (const std::exception & e)
{
 std::cerr << e.what() << std::endl;
 return EXIT_FAILURE;
}

[Shannon-Law]

@MartyG-RealSense
I'm sorry for replying you late. I have tried to generate point clouds from depth maps, using the function rs2_deproject_pixel_to_point(), as you suggested. When I run this function on my Jetson AGX Xavier, most of the coordinates are correct this time, like what I expect.

Compared to the function rs2::pointcloud::calculate(), the running speed of it is much faster than rs2_deproject_pixel_to_point(). What's more, the accuracy of point clouds generated by rs2_deproject_pixel_to_point() are lower than that of rs2::pointcloud::calculate().

Last but not least, thank you for the codes!

[MartyG-RealSense]

You are very welcome :)

RealSense team members discuss at #4315 how obtaining coordinates with rs2_deproject_pixel_to_point can be less accurate compared to doing so with points.get_vertices due to the alignment process. Advice for improving results is given at #4315 (comment)

[Shannon-Law]

@MartyG-RealSense
Okay, I'm grateful for your reply and help these days.

[MartyG-RealSense]

Hi @Shannon-Law Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[Shannon-Law]

@MartyG-RealSense Okay. I'm sorry for replying you late!