Alginment of MiDAS depth to ARCore real depth image problem

[himmetozcan]

I am trying to align (convert relative depth units to real world depth) the MiDAS depth image to ARCore depth image. I am aware that there were already closed issues on similar topics like #36 #37 #42 #43 #63 #148 and #168. I am trying to do the alignment because the ARCore depth image quality is poor and trying to improve it by using MiDAS. Below are the results for raw depth of ARCore, MiDAS, and aligned MiDAS output.

To do the alignment, I follow the steps below and this is what I understood from your paper, at Equations 1-4. Please correct me if I am wrong. The MiDAS depth image is the scaled and shifted version of real depth image. So, here I am trying to calculate these scale and shift parameters.

y = Midas depth
x = ARCore depth
s = scale param.
t = shift param.

y = s * x + t
y = Ap, where A = [[x, 1]] and p = [[s], [t]]

Now use least square to solve for p

Below is the code snippet to do this. After calculating the s, and t parameters, I do the alignment.

x = arcore_depth.copy().flatten() # ARCore Depth
y = midas_depth.copy().flatten()  # MiDAS Depth
A = np.vstack([x, np.ones(len(x))]).T
s, t = np.linalg.lstsq(A, y, rcond=None)[0]
midas_depth_aligned = (midas_depth - t) / s # Aligned MiDAS Depth 

Here I plotted the 180th column. You see after the alignment, the depth values are mostly aligned. Here what I don’t understand is the ground pixels (after 450th row), where the ARCore has like a quadratic trend, and MiDAS has a linear trend. Below I also plot the 3D point cloud versions. On the left, the ARCore, on the right the aligned MiDAS is plotted. The ground points of MiDAS output, that are closer to the camera have a weird curvature.

Below is another image, where the curvature problem is more obvious. The MiDAS output ground points have a curvature, where the ARCore ground level is flat as expected.

My first question is, am I doing the alignment properly according to your paper?

My second question is instead of a linear relationship between real depth and MiDAS depth (scale and shift), is it possible there is a non-linear relation between these two depth images?

If so, how can I do a proper alignment?

[Hazeline2018]

@himmetozcan I'm trying to correlate the real/absolute depth with the relative estimated depth obtained by proposed model so that I can derive the relationship and come up with the transform equation for real application. Although, I'm still learning the mono-depth estimation algorithm, but I too kinda think that the relationship between relative depth and absolute depth is not simply linear. It might well be on a 2nd or even 3rd order.

According to one paper I came across, they suggested a quadratic function: Y= (C0+c1X+c2X^2) *h, where c0,c1,c2 are coefficients can be obtained using least square equations, h is the camera height, and X is the relevant distance, Y is the desired real distance. I think it makes more sense. also kinda correlates better with what you are finding, the curve on the edge. Although, I'm still trying to experiment and validate that.

Just thought to drop a quick idea for your problems and findings. The paper mentioned is, in case you are interested. Please do circle back if you have new findings or breakthrough.

Taha Z, Jizat JAM. A comparison of two approaches for collision avoidance of an automated guided vehicle using monocular vision. Appl Mech Mater. 2012;145(January):547–51.

Terry

[Ano977]

@himmetozcan Would you be kind enough to explain/ share snippet of the code you are using for generating the point clouds from midas output depth map? my point cloud is stretched towards the optical center which is very confusing.

[hebangwen]

@himmetozcan Thanks for your generous sharing of this alignment code. I can get very decent visualization results by exploiting this code snippet. However, in my situation, $s$ is -39.33, while $t$ is 428.078, which means the depth of some points would be negative only if the output is greater than $t$. Have you seen this kind of result? Besides, I find you use the whole image to align MiDaS and ARCore, but I can only use a rectangle in the image, in which the depth is constant. Do these differences make the result imprecise?

[puyiwen]

I am trying to align (convert relative depth units to real world depth) the MiDAS depth image to ARCore depth image. I am aware that there were already closed issues on similar topics like #36 #37 #42 #43 #63 #148 and #168. I am trying to do the alignment because the ARCore depth image quality is poor and trying to improve it by using MiDAS. Below are the results for raw depth of ARCore, MiDAS, and aligned MiDAS output.

To do the alignment, I follow the steps below and this is what I understood from your paper, at Equations 1-4. Please correct me if I am wrong. The MiDAS depth image is the scaled and shifted version of real depth image. So, here I am trying to calculate these scale and shift parameters.

y = Midas depth x = ARCore depth s = scale param. t = shift param.

y = s * x + t y = Ap, where A = [[x, 1]] and p = [[s], [t]]

Now use least square to solve for p

Below is the code snippet to do this. After calculating the s, and t parameters, I do the alignment.

x = arcore_depth.copy().flatten() # ARCore Depth
y = midas_depth.copy().flatten()  # MiDAS Depth
A = np.vstack([x, np.ones(len(x))]).T
s, t = np.linalg.lstsq(A, y, rcond=None)[0]
midas_depth_aligned = (midas_depth - t) / s # Aligned MiDAS Depth 

Here I plotted the 180th column. You see after the alignment, the depth values are mostly aligned. Here what I don’t understand is the ground pixels (after 450th row), where the ARCore has like a quadratic trend, and MiDAS has a linear trend. Below I also plot the 3D point cloud versions. On the left, the ARCore, on the right the aligned MiDAS is plotted. The ground points of MiDAS output, that are closer to the camera have a weird curvature.

Below is another image, where the curvature problem is more obvious. The MiDAS output ground points have a curvature, where the ARCore ground level is flat as expected.

My first question is, am I doing the alignment properly according to your paper?

My second question is instead of a linear relationship between real depth and MiDAS depth (scale and shift), is it possible there is a non-linear relation between these two depth images?

If so, how can I do a proper alignment?

Thank you for your code. It is very useful for me. And I have some questions, when I align other pictures which take from one depth camera, am I still need to caculate the scale and shift again? Thank you again!

[puyiwen]

Would you be kind enough to explain/ share snippet of the code you are using for generating the point clouds from midas output depth map? my point cloud is stretched towards the optical center which is very confusing.

Hi, do you solve the problems? Is the code to align the absolute depth and Midas correct?

[puyiwen]

@himmetozcan Thanks for your generous sharing of this alignment code. I can get very decent visualization results by exploiting this code snippet. However, in my situation, s is -39.33, while t is 428.078, which means the depth of some points would be negative only if the output is greater than t. Have you seen this kind of result? Besides, I find you use the whole image to align MiDaS and ARCore, but I can only use a rectangle in the image, in which the depth is constant. Do these differences make the result imprecise?

Hi, I have read other questions about align Midas to absolute depth. And I find the author said The prediction is relative inverse depth. For each prediction, there exist some scalars a,b such that a*prediction+b is the absolute inverse depth. on question 42. If I understand correctly, it means y = kx + b, y is absolute inverse depth and x is Midas prediction. So the align code may be wrong. I write a correct code to align.
realsense_depth_array_invert = 1 / realsense_depth_array # invert absolute depth with meters
x = midas_depth_array.copy().flatten() #Midas Depth
y = realsense_depth_array_invert.copy().flatten() # Realsense invert Depth
A = np.vstack([x, np.ones(len(x))]).T
s, t = np.linalg.lstsq(A, y, rcond=None)[0]
midas_depth_aligned_invert = midas_depth_array * s + t
midas_depth_aligned = 1 / midas_depth_aligned_invert
However, my align depth still have large error from ground truth.May be the prediction of Midas is wrong, or my ground truth have wrong. Can you give me some advice? Thank you very much!!

[ShnitzelKiller]

In my own testing, I can confirm that the output of MiDaS is a scaled, shifted inverse depth AKA disparity. So if you have a target depth map, you must first invert it before computing the least squares scale and shift in disparity space.

[xieodu]

@ShnitzelKiller Hi！Gould you please tell me how to invert it before computing?