rasterizer debugging question

[wtpro]

There are many possibilities for this error. Since you are executing the code asynchronously (the default setting), the root cause may not be in this line of code, but in a previous step.

I recommend debugging with CUDA_LAUNCH_BLOCKING=1 to find out the root cause of the problem. In my experience, this is usually due to some nan variable or some unproperly allocated memory.

To be honest, I do encounter this error occasionally in our 3DGStream codebase and some other 3DGS-related repo, but it's not deterministically reproducible, so I just leave it unfixed...

Originally posted by @SJoJoK in #32

Hi,

Thanks for your amazing work!

I am trying to use the rasterizer from another work "taming 3dgs" on yours to see if the training speed can be faster.

I have encountered another illegal memory access error when trying to call the rasterizer:

File "/home/user/anaconda3/envs/3dgs/lib/python3.9/site-packages/diff_gaussian_rasterization/init.py", line 101, in forward
num_rendered, num_buckets, color, radii, geomBuffer, binningBuffer, imgBuffer, sampleBuffer, countBuffer, listBuffer, listBufferRender, listBufferDistance, centers, depths, my_radii, accum_weights, accum_count, accum_blend, accum_dist = _C.rasterize_gaussians(*args)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Since you made some changes to the rasterizer too compared to the original 3dgs raster, I would like to ask what tool did you use to debug the rasterizer during your research, since I am not able to set breakpoint on cuda/c++ code when launching the script from a python script.

Thanks a lot!

[SJoJoK]

Hi,

In fact, my changes to the rasterizer did not affect the training process, so its performance should be consistent with the original 3dgs.

In addition, because the function I implemented is relatively simple, as long as it can be compiled and passed, I didn't add breakpoints and debug... I'm not sure what tools are available. I implemented a log system with the CUDA kernel function in my previous project, but it was too complicated. I think there will be a better way, but I don't know what it is.

Thanks.

[wtpro]

Thanks for your reply!