Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix get_tile_bin_edges; revert ndc2pix #113

Merged
merged 1 commit into from
Feb 1, 2024
Merged

fix get_tile_bin_edges; revert ndc2pix #113

merged 1 commit into from
Feb 1, 2024

Conversation

liruilong940607
Copy link
Collaborator

Continue fixing #111.

It is actually not the -0.5 in ndc2pix that causes the bug -- flipping the sign only hides it. Here is another test case that still render with artifacts (bottom right corner) after #112:

import imageio
import numpy as np
import torch
from gsplat import ProjectGaussians, RasterizeGaussians

torch.manual_seed(42)

device = torch.device("cuda:0")
num_points = 3

means3d = torch.rand((num_points, 3), device=device, requires_grad=True)
scales = torch.rand((num_points, 3), device=device) * 3
quats = torch.randn((num_points, 4), device=device)
quats /= torch.linalg.norm(quats, dim=-1, keepdim=True)
rgbs = torch.rand((num_points, 3), device=device)
opacities = torch.ones((num_points, 1), device=device)

glob_scale = 0.3
viewmat = torch.eye(4, device=device)
projmat = torch.eye(4, device=device)
fx, fy = 3.0, 3.0
H, W = 256, 256
clip_thresh = 0.0
BLOCK_X, BLOCK_Y = 16, 16
tile_bounds = (W + BLOCK_X - 1) // BLOCK_X, (H + BLOCK_Y - 1) // BLOCK_Y, 1

background = torch.zeros(3, device=device)


xys, depths, radii, conics, num_tiles_hit, cov3d = ProjectGaussians.apply(
    means3d,
    scales,
    1,
    quats,
    viewmat,
    viewmat,
    fx,
    fy,
    W / 2,
    H / 2,
    H,
    W,
    tile_bounds,
)
render = RasterizeGaussians.apply(
    xys,
    depths,
    radii,
    conics,
    num_tiles_hit,
    rgbs,
    opacities,
    H,
    W,
    background,
)

canvas = (render * 255.0).detach().cpu().numpy()
imageio.imwrite("ref.png", canvas.astype(np.uint8))

ref

@liruilong940607
Copy link
Collaborator Author

In this case I trace to tile_bins which is shown to be:

tensor([[  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   1],
        [  1,   2],
        [  2,   3],
        [  3,   4],
        [  4,   5],
        [  5,   6],
        [  6,   7],
        [  7,   8],
        [  8,   9],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  9,  10],
        [ 10,  11],
        [ 11,  12],
        [ 12,  13],
        [ 13,  14],
        [ 14,  15],
        [ 15,  16],
        [ 16,  17],
        [ 17,  18],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [ 18,  19],
        [ 19,  20],
        [ 20,  21],
        [ 21,  22],
        [ 22,  23],
        [ 23,  24],
        [ 24,  25],
        [ 25,  26],
        [ 26,  27],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [ 27,  28],
        [ 28,  30],
        [ 30,  32],
        [ 32,  34],
        [ 34,  36],
        [ 36,  38],
        [ 38,  40],
        [ 40,  42],
        [ 42,  44],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [ 44,  45],
        [ 45,  46],
        [ 46,  48],
        [ 48,  51],
        [ 51,  54],
        [ 54,  57],
        [ 57,  59],
        [ 59,  61],
        [ 61,  63],
        [ 63,  65],
        [ 65,  67],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [ 67,  68],
        [ 68,  69],
        [ 69,  71],
        [ 71,  74],
        [ 74,  77],
        [ 77,  80],
        [ 80,  82],
        [ 82,  84],
        [ 84,  86],
        [ 86,  88],
        [ 88,  90],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [ 90,  91],
        [ 91,  92],
        [ 92,  94],
        [ 94,  97],
        [ 97, 100],
        [100, 103],
        [103, 105],
        [105, 107],
        [107, 109],
        [109, 111],
        [111, 113],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [113, 114],
        [114, 115],
        [115, 117],
        [117, 120],
        [120, 123],
        [123, 126],
        [126, 128],
        [128, 130],
        [130, 132],
        [132, 134],
        [134, 136],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [136, 137],
        [137, 138],
        [138, 140],
        [140, 143],
        [143, 146],
        [146, 149],
        [149, 151],
        [151, 153],
        [153, 155],
        [155, 157],
        [157, 159],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [159, 160],
        [160, 161],
        [161, 163],
        [163, 166],
        [166, 169],
        [169, 172],
        [172, 174],
        [174, 176],
        [176, 178],
        [178, 180],
        [180, 182],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [182, 183],
        [183, 185],
        [185, 187],
        [187, 189],
        [189, 191],
        [191, 193],
        [193, 195],
        [195, 197],
        [197, 199],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [199, 200],
        [200, 202],
        [202, 204],
        [204, 206],
        [206, 208],
        [208, 210],
        [210, 212],
        [212, 214],
        [214, 216],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [216, 217],
        [217, 218],
        [218, 219],
        [219, 220],
        [220, 221],
        [221, 222],
        [222, 223],
        [223, 224],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [  0,   0],
        [224, 225],
        [225, 226],
        [226, 227],
        [227, 228],
        [228, 229],
        [229, 230],
        [230,   0],
        [  0, 232]], device='cuda:0', dtype=torch.int32)

The last two rows are buggy and they are the reason of this artifacts.

@liruilong940607
Copy link
Collaborator Author

liruilong940607 commented Jan 31, 2024 •
edited
Loading

Also this artifacts is introduced in commit bae8738, which helps locate to the get_tile_bin_edges function.

After this fix the last two rows of the tile_bins becomes:

[230, 231],
[231, 232]], device='cuda:0', dtype=torch.int32)

which eliminates the artifacts:

image

@liruilong940607 liruilong940607 requested a review from vye16 January 31, 2024 23:18
@liruilong940607
Copy link
Collaborator Author

So I revert the +/- 0.5 in #112

vye16
vye16 approved these changes Feb 1, 2024
View reviewed changes
Copy link
Collaborator

@vye16 vye16 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@vye16 vye16 merged commit e13eff4 into main Feb 1, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vye16 vye16 vye16 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@liruilong940607 @vye16