Color map optimization #230 #339

syncle · 2018-05-04T09:21:16Z

This resolves #230

Example:

syncle

Minor comment. Additional review is required.

syncle · 2018-05-04T10:35:26Z

src/Core/Geometry/Image.cpp

+    return (u >= inner_margin && u < width_ - inner_margin &&
+        v >= inner_margin && v < height_ - inner_margin);
+}
+


I add this as it is frequently used in this PR. However, this function also can be in Geometry2D.h. Not quite sure this is the right place.

I think it is a good place.

syncle · 2018-05-04T10:36:19Z

src/Core/Utility/Eigen.cpp

-        return std::make_tuple(solution_exist, std::move(x));
+    if (check_det) {
+        bool solution_exist = true;
+        double det = A.determinant();


This is quite bad idea for large scale matrix. I add check_det option here.

syncle · 2018-05-04T10:36:56Z

src/Core/Utility/Eigen.h

@@ -53,7 +53,7 @@ Eigen::Vector6d TransformMatrix4dToVector6d(const Eigen::Matrix4d &input);

 /// Function to solve Ax=b
 std::tuple<bool, Eigen::VectorXd> SolveLinearSystem(
-    const Eigen::MatrixXd &A, const Eigen::VectorXd &b);
+    const Eigen::MatrixXd &A, const Eigen::VectorXd &b, bool check_det = true);


it checkes determinant by default. Hence, it should not affect original pipeline.

syncle · 2018-05-04T10:40:08Z

src/Python/Core/py3d_registration.cpp

+			&TextureMapOptimization,
+			"Function for optimizing texture mapping for reconstructed scenes",
+			"mesh"_a, "imgs_rgbd"_a, "camera"_a,
+            "option"_a = TextureMapOptmizationOption());


I put texture_map_optimization in registration module. Not quite sure this is the right place.

syncle · 2018-05-04T10:43:28Z

I found a minor bug in this PR. Working on now.

syncle · 2018-05-04T23:09:02Z

src/Python/Core/py3d_registration.cpp

+#include <Core/Geometry/RGBDImage.h>
+#include <Core/Geometry/TriangleMesh.h>
+#include <Core/Registration/ColorMapOptimization.h>
+#include <Core/Camera/PinholeCameraTrajectory.h>


Can we place ColorMapOptimization under py3d_registration? Can add these headers? Maybe this would not match the category.

I think color map optimization is a very good feature. We should highlight it. Registration is specifically about aligning point clouds. Color map optimization is more about putting color on a mesh. I think it is worth a separate folder. (At least it is at the same or larger scale of Integration or Odometry.)

syncle · 2018-05-04T23:11:28Z

src/Python/Tutorial/Advanced/color_map_optimization.py

+sys.path.append("../Utility")
+from common import *
+
+path = "[set_this_path_to_fountain_dataset]"


I didn't include dataset for ColorMapOptimization. Instead, I had added google drive link to download the dataset. The file size for this tutorial is about 30MB (including mesh, 33 color and depth image pairs), which is not need to be maintained in Github repo.

qianyizh · 2018-05-05T07:04:38Z

Wow, this is awesome! Let me review it during the weekend.
I have been dreaming to have it in Open3D for years!!

qianyizh · 2018-05-16T07:47:00Z

I have some non-trivial comments on this. So the review process is slow.
I will try to finish it asap.

qianyizh

This is amazing. My main issue is that we should move this out from Registration. Registration should be preserved for only 3d-to-3d alignment. RGB-D odometry is for RGB-D-to-RGB-D alignment, and color map optimization is kinda RGB-D-to-3D alignment. So it is totally worth a separate folder. Plus, this is a very useful function. In the future, we should also consider extend it with RGB-to-3D alignment.

Another minor comment is that we should also test on a sequence of a 360 scan. It is a pretty common use case, and can test if there is ghosting artifacts. See my detailed comments.

qianyizh · 2018-05-12T02:09:17Z

src/Core/Geometry/Image.cpp

+    return (u >= inner_margin && u < width_ - inner_margin &&
+        v >= inner_margin && v < height_ - inner_margin);
+}
+


I think it is a good place.

qianyizh · 2018-05-12T02:10:19Z

src/Core/Utility/Eigen.cpp

-        return std::make_tuple(solution_exist, std::move(x));
+    if (check_det) {
+        bool solution_exist = true;
+        double det = A.determinant();


qianyizh · 2018-05-12T02:10:25Z

src/Core/Utility/Eigen.h

@@ -53,7 +53,7 @@ Eigen::Vector6d TransformMatrix4dToVector6d(const Eigen::Matrix4d &input);

 /// Function to solve Ax=b
 std::tuple<bool, Eigen::VectorXd> SolveLinearSystem(
-    const Eigen::MatrixXd &A, const Eigen::VectorXd &b);
+    const Eigen::MatrixXd &A, const Eigen::VectorXd &b, bool check_det = true);


qianyizh · 2018-05-12T02:13:24Z

src/Python/Core/py3d_registration.cpp

+#include <Core/Geometry/RGBDImage.h>
+#include <Core/Geometry/TriangleMesh.h>
+#include <Core/Registration/ColorMapOptimization.h>
+#include <Core/Camera/PinholeCameraTrajectory.h>


I think color map optimization is a very good feature. We should highlight it. Registration is specifically about aligning point clouds. Color map optimization is more about putting color on a mesh. I think it is worth a separate folder. (At least it is at the same or larger scale of Integration or Odometry.)

qianyizh · 2018-05-12T02:13:47Z

src/Python/Tutorial/Advanced/color_map_optimization.py

+sys.path.append("../Utility")
+from common import *
+
+path = "[set_this_path_to_fountain_dataset]"


qianyizh · 2018-06-22T07:55:16Z

src/Core/Registration/ColorMapOptimization.cpp

+        MakeVertexAndImageVisibility(const TriangleMesh& mesh,
+        const std::vector<RGBDImage>& images_rgbd,
+        const PinholeCameraTrajectory& camera,
+        const ColorMapOptmizationOption& option)


I think back when I made visibility test, I mask all vertices around depth discontinuity as invisible. This is because at depth discontinuity, the depth image becomes noisy, the measurement is not accurate in terms of both depth and x-y value. The projection also gets unstable. This happens frequently when you do a 360 scan of an object. So I chose to mask visible vertices only when:

projected depth is close to the depth in depth image

the point on depth image should have a normal close to the viewing ray

Can you try a sequence which is a 360 scan and see if you see any boundary ghosting? If so, the pixels around depth discontinuity need to be eroded.

This also applies when averaging pixel values. Only points with a good normal orientation (not too far away from the viewing ray) gets averaged.

qianyizh · 2018-06-26T15:37:23Z

@syncle The build is broken. Can you check?

syncle · 2018-06-26T16:09:23Z

Got it. I am working on this now. This PR is also needed to be tested on the suggested case.

syncle

Self review on the revised version.

syncle · 2018-06-29T02:15:48Z

src/Core/Geometry/Image.cpp

+                }
+            }
+        }
+    }


I am not quite like this function. All the filtering functions in Open3D are implemented using 1D domain to save computational burden, but for this function, I had to use 4 nested for-loops. Maybe there would be better implementation for this.

I think it is fine for now. In the long run, we should seriously consider how to use OpenCV in Open3D. Actually I think vcpkg might be our savior. All that is missing is a package management system on Windows. Thus we had to include all the source code and are afraid of linking to big packages like OpenCV. But if Windows have a package management system, lots of our problems will be solved. We should invest time in this. @syncle @takanokage

syncle · 2018-06-29T02:18:54Z

src/Python/Core/open3d_colormap.cpp

+            "Function for color mapping of reconstructed scenes via optimization",
+            "mesh"_a, "imgs_rgbd"_a, "camera"_a,
+            "option"_a = ColorMapOptmizationOption());
+}


I am not sure color map is good name for this module. We already uses color map name for vertex/mesh coloring. However, I had to use this name again because texture_map_optimization may lead users to think some 2D texture map representations.

Yeah it is a bit confusing. But I can't come up with better names. Let's just use this one for now.

syncle · 2018-06-29T02:20:03Z

src/Python/Tutorial/Misc/sampling.py

+            cnt += 1
+    copyfile(path + "/scene/cropped.ply",
+            out_path + "/scene/integrated.ply")
+    write_trajectory(traj, out_path + "scene/key.log")


This python script just samples every sampling_rate frames from the RGBD sequence. Actually it would be good this script can select only sharp frames.

Yes. This is a good point. A simple fix is to use cv2.Sobel to generate the magnitude of image gradient, then sum it to estimate the sharpness. Then use a rolling window to pick the sharpest frames once in a while.

syncle · 2018-06-29T02:20:41Z

Few changes:

Implement a function to generate depth discontinuity mask
Image dilation function in Image class
Some misc scripts that can be used with Reconstruction system
Tested with newly captured images

Color image

Depth discontinuity mask

Before optimization

After optimization

qianyizh · 2018-07-01T05:50:54Z

Looks good. Merging.

lrxjason · 2018-07-15T05:52:28Z

Is it normal for the Residual error increase in some iteration and decrease later?
[Iteration 0001] Residual error : 17140.453393, reg : 0.000000
[Iteration 0002] Residual error : 15428.754199, reg : 37.112254
[Iteration 0003] Residual error : 14066.208086, reg : 92.925209
[Iteration 0004] Residual error : 12799.874035, reg : 159.233673
[Iteration 0005] Residual error : 11881.530189, reg : 224.982045
[Iteration 0006] Residual error : 10978.509944, reg : 291.869497
[Iteration 0007] Residual error : 10164.817606, reg : 357.346255
[Iteration 0008] Residual error : 9418.470241, reg : 419.045432
[Iteration 0009] Residual error : 8753.664132, reg : 471.310913
[Iteration 0010] Residual error : 8271.234603, reg : 523.733466
[Iteration 0011] Residual error : 7810.671108, reg : 574.601559

[Iteration 0012] Residual error : 7503.250006, reg : 623.425672
[Iteration 0013] Residual error : 7182.370144, reg : 668.310040
[Iteration 0014] Residual error : 7944.772004, reg : 3436.285845
[Iteration 0015] Residual error : 8398.769080, reg : 1366.687143
[Iteration 0016] Residual error : 8075.550004, reg : 1086.460814
[Iteration 0017] Residual error : 7834.747559, reg : 1121.454900
[Iteration 0018] Residual error : 7645.106373, reg : 1087.304552
[Iteration 0019] Residual error : 7586.346190, reg : 1110.170485
[Iteration 0020] Residual error : 7510.479685, reg : 1142.089592
[Iteration 0021] Residual error : 7360.104927, reg : 1175.957038

[Iteration 0022] Residual error : 7256.626958, reg : 1210.944181
[Iteration 0023] Residual error : 7166.587797, reg : 1246.043558
[Iteration 0024] Residual error : 7113.245672, reg : 1324.791503
[Iteration 0025] Residual error : 7013.289640, reg : 1359.360750
[Iteration 0026] Residual error : 6900.794533, reg : 1392.757199
[Iteration 0027] Residual error : 6926.551166, reg : 1339.472581
[Iteration 0028] Residual error : 6869.239273, reg : 1354.990521
[Iteration 0029] Residual error : 6785.210600, reg : 1379.802116
[Iteration 0030] Residual error : 6785.631435, reg : 1411.250541
[Iteration 0031] Residual error : 6757.596825, reg : 1582.067014
[Iteration 0032] Residual error : 6721.654318, reg : 1448.798946
[Iteration 0033] Residual error : 6709.315793, reg : 1454.303839
[Iteration 0034] Residual error : 6604.872329, reg : 1470.278581

syncle · 2018-07-15T06:05:02Z

That may happen when images are being bended too much. You can notice reg increases suddenly for this reason. In that case, you can make iteration more conservative way - making anchor point more stubborn. Consider to increase option.non_rigid_anchor_point_weight.

lrxjason · 2018-07-16T04:08:41Z

Thank you very much. After the increase, the error decreased more stable. @syncle
After color optimization, Some parts of the surface is black which has color in the non-optimized mesh. Is there any parameter to deal with that?

syncle · 2018-07-16T06:00:37Z

Consider turn off non_rigid_camera_coordinate. In case of non-rigid image deformation, there is a the case if a image is warped inwards.

chunduri11 · 2020-12-16T21:25:42Z

Do you have any plans to consider implementing alternating optimisation for Color map optimisation?
This could give significant speed up with out compromising on the quality of the results.

theNded · 2020-12-17T01:18:31Z

Do you have any plans to consider implementing alternating optimisation for Color map optimisation?
This could give significant speed up with out compromising on the quality of the results.

I have commented that in #2686.

chunduri11 · 2020-12-17T09:28:09Z

Thanks, this information Is very useful.

Adding color map optimization

37baaa2

syncle requested a review from qianyizh May 4, 2018 09:25

syncle commented May 4, 2018

View reviewed changes

merge with master, change name as ColorMapOptimization

82a5ae3

syncle force-pushed the texture_map_optimization branch 3 times, most recently from 016ab36 to 4e5f37d Compare May 4, 2018 22:39

syncle commented May 4, 2018

View reviewed changes

syncle added this to the v0.2 milestone May 9, 2018

syncle added the enhancement label May 9, 2018

syncle self-assigned this May 9, 2018

bug fix for ColorMapOptimization

2110775

syncle force-pushed the texture_map_optimization branch from 4e5f37d to 2110775 Compare May 11, 2018 15:46

syncle changed the title ~~Color map optimization~~ Color map optimization #230 May 15, 2018

syncle added 2 commits May 18, 2018 16:53

minor fix for ColorMapOptimization & merge with master

4a9575b

Merge branch 'appveyor_fix' into texture_map_optimization

76d5893

qianyizh reviewed Jun 22, 2018

View reviewed changes

Merge branch 'master' into texture_map_optimization

946ac53

move ColorMapOptimization files & implement image dilation

aa6c266

syncle force-pushed the texture_map_optimization branch from cc44915 to aa6c266 Compare June 26, 2018 16:07

qianyizh mentioned this pull request Jun 26, 2018

[Question] denoising point cloud and mesh reconstruction #410

Closed

syncle added 3 commits June 26, 2018 12:16

Merge branch 'master' into texture_map_optimization

71a8be4

Fix issues on color_map_optimization.rst

43b6d7b

bug fix regarding ColorMapOptmization definitions

8292024

syncle added 2 commits June 28, 2018 15:43

minor bug fixes in DilateImage

1ce86ee

minor bug fixes in ColorMapOptimization.cpp

909f5ae

syncle force-pushed the texture_map_optimization branch 3 times, most recently from 549df4c to a0d0c05 Compare June 29, 2018 02:14

syncle commented Jun 29, 2018

View reviewed changes

removing unused variables

0f61c65

syncle force-pushed the texture_map_optimization branch from a0d0c05 to 0f61c65 Compare June 29, 2018 04:05

qianyizh merged commit dd511a0 into isl-org:master Jul 1, 2018

syncle deleted the texture_map_optimization branch July 5, 2018 07:33

Color map optimization #230 #339

Color map optimization #230 #339

Conversation

syncle commented May 4, 2018

syncle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

syncle commented May 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qianyizh commented May 5, 2018

qianyizh commented May 16, 2018

qianyizh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qianyizh commented Jun 26, 2018

syncle commented Jun 26, 2018

syncle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

syncle commented Jun 29, 2018

Color image

Depth discontinuity mask

Before optimization

After optimization

qianyizh commented Jul 1, 2018

lrxjason commented Jul 15, 2018

syncle commented Jul 15, 2018

lrxjason commented Jul 16, 2018

syncle commented Jul 16, 2018

chunduri11 commented Dec 16, 2020

theNded commented Dec 17, 2020

chunduri11 commented Dec 17, 2020

syncle commented May 4, 2018 •

edited

Loading