JOSS: 3D reconstruction

[mhubii]

I have successfully detected all particles. Then imported:

• Camera Calibration: /tmp/ParticleTracking/RodTracker/src/RodTracker/resources/example_data/calibrations/gp34.json
• World Transformation: /tmp/ParticleTracking/RodTracker/src/RodTracker/resources/example_data/calibrations/transformation.json

I then hit Solve, but nothing happens? Could you explain a little better how to reconstruct. What am I doing wrong?

When Update Plots, this shows up

This seems also related to this issue: #96

Refers to openjournals/joss-reviews#5986

[dmitrypuzyrev]

Hi @mhubii,

thank you, I am working on this issue, I think it might be something different to #96.
That one was already solved in the stable release. Unfortunately, @merrygoat did not check it in his setup and did not reply yet.

Could you please let me know what you did differently to @merrygoat :

• you ran identification for all particle colors at once, not only for one color, right?
• did you correct the rod positions and add missing (non-detected automatically) rods before solving?

[mhubii]

shouldn't the 3D reconstruction work regardless of detecting all particles?

[dmitrypuzyrev]

Hi @mhubii,
yes, it should work. The results won't be very accurate, but it should not give an error.

I am trying to reproduce your problem but cannot so far.
I have loaded images from gp3 folder to Front view, then gp4 to Top view, then ran detection for all particle colors without any changes or corrections.
I have then loaded same stereo calibration and coordinate transformation files.
Is it exactly what you do here?

For me it works as then intended if I press solve, both with "Tracking" checkbox marked and unmarked.

Please check the attached screenshot, the detected rod data looks identical to yours, but I cannot compare the cropped part of your image:

[dmitrypuzyrev]

Hi again @mhubii,

I think I got it now, you try to do 3D reconstruction after loading images from only one camera, it is impossible in this setup)
So it is also related to issue #110.

I guess it's a problem of documentation. I will update it ASAP and also add corresponding warning messages to Solve button.

[mhubii]

hm I see, yes I probably only loaded gp3. Given the segmentations and rods of known dimension, you should be able to solve pnp regardless. But I suppose this would be a separate issue

[dmitrypuzyrev]

Hi @mhubii,
It is another issue indeed.
We don't think it would be realistic to have enough information for a proper 3D reconstruction from one camera view in case of rods. The main problem is that one cannot really infer how far the rod is from camera based on its visible length, since it depends on the unknown angle between the line from the camera to the rod and the rod main axis; and the visible width of the rods do not change significantly enough to help to infer this distance.
Single view-based reconstructions were used in some experiments with the spherical particles, I guess in case of thicker cylindrical objects it might be possible to a degree, but not in case of relatively thin rods, unfortunately.

[mhubii]

I mean you've got a rod of known length so classifying 4 points (e.g. 0/3, 1/3, 2/3, 3/3 x length) on it would allow you to solve PnP (I might be wrong, but am somewhat confident this should work). Please refer e.g. to OpenCV https://docs.opencv.org/4.x/d5/d1f/calib3d_solvePnP.html

However, you might be right that there are redundant solutions for a rod. I think I understand now. Thanks for clarifying.

[dmitrypuzyrev]

Could it be that you were initially confusing the PnP problem with its inverse problem, triangulation/reconstruction?
In our case the camera position is known, and we need to transform coordinates from camera frame to world frame, not the other way around. Generally, this problem cannot be solved with one camera view (unless one has a camera with depth information, like LIDAR), one either needs a stereo camera or moving (scanning) camera.

In some cases it can be solved with a single view. See, for example, here:
https://cvg.cit.tum.de/research/image-based_3d_reconstruction/singleviewreconstruction
Unfortunately, in case of thin rods it does not seem applicable.

[mhubii]

well inverse or not, kind of the same problem. You can estimate poses simply using AprilTags or Aruco markers using PnP, no need for depth, just an object of known shape and correspondences in the image space. I don't quite understand the need for a world frame here, from my understanding you just need a reference frame. You can even go down the alley of differentiable rendering or what not in your case. But I think this is a little beyond.

[dmitrypuzyrev]

Hi @mhubii,

you are completely right that AprilTags or Aruco markers are a very good idea. For example, it would help with long-time tracking of particles or detection of some rotational motions which are otherwise hard to access. We thought about it for possible experiments with larger particles, in this case, however, the size constrains do not allow it (the rods are actually less than 1mm thick).