undistortion for fisheye images is not Right?

[scott198510]

I found that the code for removing distortion from fish eyes has been changed several times, but I still find it to be incorrect。

x1 = (grid_x - cx) / fx y1 = (grid_y - cy) / fy theta = np.sqrt(x1**2 + y1**2) r = ( 1.0 + params[0] * theta**2 + params[1] * theta**4 + params[2] * theta**6 + params[3] * theta**8 ) mapx = (fx * x1 * r + width // 2).astype(np.float32) mapy = (fy * y1 * r + height // 2).astype(np.float32)

My code:
x1 = (grid_x - cx) / fx y1 = (grid_y - cy) / fy r = np.sqrt(x1**2 + y1**2) theta = np.arctan2(r,1) rd = (1.0 *theta + params[0] * theta**3 + params[1] * theta**5 + params[2] * theta**7 + params[3] * theta**9) scale = rd/r if r!=0 else 1.0 mapx = (fx * x1 * scale + width // 2).astype(np.float32) mapy = (fy * y1 * scale + height // 2).astype(np.float32)

[scott198510]

@akanazawa @maturk

[akanazawa]

thanks pinged relevant people who can check this

[jefequien]

In this context, undistortion is a bit of a misnomer. We want to remap distorted fisheye images to ideal fisheye images to retain FOV. Looks like you are following the equations from OpenCV's website here. Those are the equations for remapping distorted fisheye images to ideal pinhole images, which is not what we want.

This implementation is a numpy version of the preprocessing used by the original Fisheye-GS repository.
https://github.com/zmliao/Fisheye-GS/blob/master/prepare_scannetpp.py#L51

[scott198510]

@jefequien
remap distorted fisheye images to ideal fisheye images？What is the difference of ideal fisheye images and distorted fisheye images,
Usually, when we talk about fisheye images, there is obvious distortion. We do not usually refer to ideal fisheye or distorted fisheye, nor do we refer to equiaxed projection as ideal fisheye. I can only assume that fisheye-gs used this equiaxed projection method in this paper, and the subsequent algorithms are based on the Jacobian matrix obtained from this projection. I tried changing it to the kind of image I mentioned earlier, which was transformed into a completely pinhole camera, and the training resulted in a completely blurry result

[scott198510]

In this context, undistortion is a bit of a misnomer. We want to remap distorted fisheye images to ideal fisheye images to retain FOV. Looks like you are following the equations from OpenCV's website here. Those are the equations for remapping distorted fisheye images to ideal pinhole images, which is not what we want.

This implementation is a numpy version of the preprocessing used by the original Fisheye-GS repository. https://github.com/zmliao/Fisheye-GS/blob/master/prepare_scannetpp.py#L51

@jefequien

When I first saw the de distortion code provided in his link, I was stunned. At first glance, I thought its undistortion code was wrong. Why could he still get the rendering results that looked good in the paper

[jefequien]

An ideal fisheye camera uses equidistant projection. An ideal pinhole camera uses rectilinear projection. If you collect a dataset with a wide-angle lens, you can undistort it to rectilinear image and train the splat normally but you'd be throwing away a lot of FOV. With Fisheye-GS, you can undistort it to an equidistant image and train a splat with a wider FOV.

[scott198510]

@jefequien

With Fisheye-GS, you can undistort it to an equidistant image and train a splat with a wider FOV.
Yes, as you said, I have compared the training results of these two methods after distortion removal. The FOV obtained by fisheye gs is significantly larger, but I have a question.
This method is also as currently written in the gsplat code. The four corners of the original image are still lost with a small amount of information. If I want to obtain the same size as the original image after isometric projection, that is, retain more original image information, how can I modify it appropriately? Do you have any good suggestions?

[jefequien]

All of the information in the original image is retained. The black areas in the corners were never captured. Since the corners lack supervision, if you find them distracting, you can crop them away after rendering.

Or you can try to invert the undistortion. Undistortion is hard to invert because it is not one-to-one, (see the 8-degree polynomial). However, it looks one-to-one so it feels like it should be invertable. I've looked into it before but haven't found a satisfactory solution. See stackoverflow discussions.
https://stackoverflow.com/questions/67008814/calculate-and-use-inverse-of-opencv-distortion-parameters
https://stackoverflow.com/questions/21615298/opencv-distort-back

[scott198510]

@jefequien Thanks.
But you probably don't fully understand what I meant. What I meant was to expand the mapx and mapy a bit, that is to say, when using equiaxed projection as fisheye-gs method to remove distortion, I need to expand the range of the projected image a bit. This way, the range of the ROI obtained after crop will still be the same size as the original image.
It is obvious that the roi_undist range in the following code is smaller than the size of the original image。

            mapx = (fx * x1 * r + height // 2).astype(np.float32)
            mapy = (fy * y1 * r + height // 2).astype(np.float32)

            # Use mask to define ROI
            mask = np.logical_and(
                np.logical_and(mapx > 0, mapy > 0),
                np.logical_and(mapx < width - 1, mapy < height - 1),
            )
            y_indices, x_indices = np.nonzero(mask)
            y_min, y_max = y_indices.min(), y_indices.max() + 1
            x_min, x_max = x_indices.min(), x_indices.max() + 1
            mask = mask[y_min:y_max, x_min:x_max]
            K_undist = K.copy()
            K_undist[0, 2] -= x_min
            K_undist[1, 2] -= y_min
            roi_undist = [x_min, y_min, x_max - x_min, y_max - y_min]

If we directly call OpenCV functions instead of implementing equidistant projection with handwritten code, you can do it this way, but this is not the effect I want. What I get is a completely distortion free effect, similar to a pinhole projection image

w = int(img.shape[1])
h = int(img.shape[0])
border_width  = int(w/4)
border_height = int(h/4)

img_bordered = cv2.copyMakeBorder(img, border_height, border_height, border_width, border_width, cv2.BORDER_ISOLATED)
h_new, w_new = img_bordered.shape[:2]

new_camera_matrix1, roi = cv2.getOptimalNewCameraMatrix(camera_matrix, dist_coeffs[:4], (w_new, h_new), 0.5, (w, h))

map1, map2 = cv2.fisheye.initUndistortRectifyMap(camera_matrix, dist_coeffs[:4], np.eye(3), new_camera_matrix1, (w_new, h_new), cv2.CV_16SC2)

undistort_img = cv2.remap(img_bordered, map1, map2, cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)