Depth Stream Distortion D400 Series

[vladsterz]

Required Info
Camera Model	D400
Firmware Version	Latest
Operating System & Version	Win 10
Platform	PC
SDK Version	Latest
Language	C++
Segment	Human Reconstruction

Issue Description

Hello once again.Today I want to tackle the problem of distortion in the depth "sensor".

The application that I work on aims to reconstruct a person from a multi-view perspective (Poisson reconstruction to be accurate). We use 4 cameras in order to capture the full human point-cloud and our cameras are flipped horizontally ( Important If depth senors do suffer from radial distortion, flipping camera horizontally yields the danger that head and feet are going to be severely distorted as they are too far from the center of the lens).
For calibration we use this work, which works pretty fine in general.

So far, the (merged) point cloud I get is indeed a mess in the head area.
The awkward moment comes here. In a moment of despair I tried to un-distort the depth image using this code, with radial distortion coefficients chosen pretty much in luck. After some fine-tuning I managed to get a much prettier result in the head region.

As my research has shown, IR cameras are calibrated with some black magic Intel-RealSense method/software, not publicly available. Despite the fact that IR sensor DO have a distortion model, their distortion parameters are set to zero (0).

So, has anyone faced the same, or even similar, problem as well? How did you overcome it? Un-distortion in stereo images is not that trivial, since the depth is calculated internally in the device. The optimal solution would be un-distorting the IR sensors, but I am too poor for the calibration structure provided from Intel :( .

**NOTE 1 : ** Unfortunately I left the office and cannot provide the screenshots to support my aforementioned arguments. Maybe if you show interest in this topic, I will provide it at Monday.
**NOTE 2 : ** Using D-415
**NOTE 3 : ** Feel free to ask about anything about the project I am working on.

[MartyG-RealSense]

Although calibration of multiple cameras is undoubtedly easier with the $1500 system in the Intel Click store, it is possible to put your own home made calibration together using the open source software Vicalib.

In Intel's multiple camera webinar, it was stated about Vicalib that "It uses a board that you show to each of the cameras in turn., and it establishes overlapping regions to then minimize the pose of each of those together".

https://github.com/arpg/vicalib

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @vladsterz,

Indeed, depth generated from D4 ASIC is rectified. You can also try to use 'customRW' tool in the dynamic calibration tool to read the distortion model. D415 unrectified L/R IR only supports 1920x1080 and Y16 format. D435 unrectified L/R IR only support 1280x800 and Y16 format as the custom calibration white paper mentioned.
https://downloadcenter.intel.com/download/27955/Intel-RealSense-D400-Series-Calibration-Tools-and-API

[vladsterz]

Hello @RealSense-Customer-Engineering , I totally didn't expect an answer to this question after all this time, thanks for coming back to it.
I've messed around with this tool actually, but still didn't manage to find a good solution. I've a question. Do you think that estimating distortion coefficients, focal lengths and principal points using an external tool and updating the devices' with these estimated values would indeed give a better result? If not, please explain why.
Thanks in advance

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @vladsterz,

To use an external tool which has a better calibration algo to achieve a required depth quality criteria is good. Our OEM calibration tool which requires our target chart defines our own criteria. Once you calibrate and update the parameters in the D415 device, you can utilize our DQT tool to evaluate how the depth accuracy and fill rate can achieve and compare with ours. You also can consider to calibrate RGB distortion model which we don't provide.

[RealSenseCustomerSupport]

---

I will close this if no more help required.

[vladsterz]

Actually it is a frozen topic right now because of other jobs. Still, I am gonna get back to this issue.