Needing clarification for implementation of Inverse-Brown-Conrady transformation

[sisiplac]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	L515 and D455
Firmware Version	01.05.00.00 (L515) and 05.12.08.200 (D455)
Operating System & Version	Win 10
Platform	PC
SDK Version	2.39
Language	C/C#
Segment	AR/Calibration

Issue Description

We are using RealSense L515 devices and are trying to fetch the color intrinsics to do some Augmented-Reality visualizations.
The color intrinsics of the L515 devices are given as Inverse-Brown-Conrady transformation.
The implementation of the Inverse-Brown-Conrady projection and deprojection functions seems very strange to me, due to the following reasons:

• For Inverse Brown Conrady, there is a closed form solution for both the projection and deprojection code available, see methods rs2_project_point_to_pixel and rs2_deproject_pixel_to_point in https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense2/rsutil.h
  According to the documentation (https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20) the projection part should need iteration or lookup tables, what corresponds with my knowledge about the Inverse Brown Conrady transformation.
• When calling the projection part with a point (a, b, 1) we obtain a projected point (c, d). Calling then the deprojection function on (c, d) with depth 1 we do NOT get back (a, b, 1). Instead, there is a significant offset compared to the original point. That means, that either the projection or deprojection code must be wrong.
• The projection code for Inverse-Brown-Conrady and Modified-Brown-Conrady is absolutely identical, what is also strange. Why are two different model names used for the same mathematical transformation?

From my point of view the parameters for the L515 seem to follow the Modified-Brown-Conrady or Brown-Conrady model, but NOT the Inverse-Brown-Conrady model. We were able to confirm this by doing a calibration of the L515 color channel using our own tools and comparing this transformation with the parameters obtained by the SDK.

What I would recommend to fix the problem:

• Remove the RS2_DISTORTION_INVERSE_BROWN_CONRADY path from the rs2_project_point_to_pixel path and include assert(intrin->model != RS2_INVERSE_DISTORTION_BROWN_CONRADY) instead
• Return the model Modified-Brown-Conrady or Brown-Conrady for the color channel of L515 devices.
  Important question: Which of the two models is used?

[sisiplac]

Has no one else stumbled across the inconsistent projection and deprojection for Inverse-Brown-Conrady?

[delmottea]

I got the same problem as you.
I initially tried to match between undistorted features from color channel (using inverse brown conrady model as specified from the device) to the reprojected points from the depth camera (rs2_deproject_pixel_to_point -> rs2_transform_point_to_point) but the matching returns a large error.
By reprojecting the points from the depth camera (rs2_deproject_pixel_to_point -> rs2_transform_point_to_point -> rs2_project_point_to_pixel), it works fine, which seems to indicate that the model used for the l515 color stream is a Modified-Brown-Conrady or Brown-Conrady instead of the Inverse-Brown-Conrady returned by the device.

It would be good to have an official confirmation of which model has been used for the calibration (including the exact formula used) and that the erroneous code get removed from the lib.

[sisiplac]

Further important observation:
The new RealSense D455 device is showing exactly the same problem:

• Color intrinsics show model "InverseBrownConrady" (and distortion coefficient are non-zero).
• The real model for the D455 color intrinsics seems to be Modified-Brown-Conrady or Brown-Conrady. It would be very important to know which model the parameters represent (Modified-Brown-Conrady or Brown-Conrady?).

At least our D435 model seems to be not affected by this bug, since all radial and tangential distortion coefficients of the color intrinsics are zero, so it does not matter, whether the model is Brown-Conrady, Inverse-Brown-Conrady or Modified-Brown-Conrady.

[cdb0y511]

Hi, @sisiplac . I glad someone finally finds the Inverse-Brown-Conrady is something wrong. I think this is the problem that causes the projection and deprojection a few pixels away. I have to use alignment to depth frame to avoid this problem.
So, if I still align to the RGB frame. the camera model should be

 if (intrin->model == RS2_DISTORTION_BROWN_CONRADY)
{
    float r2 = x * x + y * y;
    float f = 1 + intrin->coeffs[0] * r2 + intrin->coeffs[1] * r2*r2 + intrin->coeffs[4] * r2*r2*r2;

    float xf = x * f;
    float yf = y * f;

    float dx = xf + 2 * intrin->coeffs[2] * x*y + intrin->coeffs[3] * (r2 + 2 * x*x);
    float dy = yf + 2 * intrin->coeffs[3] * x*y + intrin->coeffs[2] * (r2 + 2 * y*y);

    x = dx;
    y = dy;
}

Right?

However, the deproject for MODIFIED_BROWN_CONRADY is missing
assert(intrin->model != RS2_DISTORTION_MODIFIED_BROWN_CONRADY); // Cannot deproject from a forward-distorted image

How do you fix it?

[sisiplac]

Yes, deprojection is missing for MODIFIED_BROWN_CONRADY.

To get an invertible transformation, you could do the following:

1. Use the "project"-function to compute arbitrary 3D/2D correspondences for a planar target in different poses.
2. Use these correspondences as input for a camera calibration routine (e.g. OpenCV) and estimate the standard (non-modified) model parameters.
3. OpenCV should have both projection- and deprojection-methods for the standard model.

Alternatively, you could also implement your own nonlinear estimator for the MODIFIED_BROWN_CONRADY deprojection. Compared to the first option this is probably more complex and more prone to errors.

[5trobl]

For that deprojection you might find the Matlab code by @diegonehab and me useful: https://www.dlr.de/rm/en/Portaldata/52/Resources/Software/CalLab/matlab_f_undistort_triangulate.zip
(from www.robotic.de/callab)

I read somewhere that they are not supporting RGB undistortion for D400, that might be the reason/consequence for/of its absence. In addition, they usually map depth frames into the rgb/color frames, and not the other way around (i.e., rgb deprojection is rarely needed).

[RealSenseSupport]

Hi,

While engineering continues to work on this, they have provided the following PR to help mitigate these issues.

#8523

Thanks

[RealSenseSupport]

Hi,

Will you be needing further help with this?

Thanks

[RealSenseSupport]

Hi,

This has been implemented in the following PR:

#8759

Will you be needing any further help with this?

If we don’t hear from you in 7 days, this issue will be closed.

Thanks