added influence pruning, depth rendering; fixed encoding/decoding bugs

README.md

@@ -1,10 +1,11 @@
 # EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS
+### ECCV 2024
 [Sharath Girish](https://sharath-girish.github.io/), [Kamal Gupta](https://kampta.github.io/), [Abhinav Shrivastava](https://www.cs.umd.edu/~abhinav/) <br><br>
 ![Teaser image](assets/teaser.png)
 
 Official implementation of the paper "EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS"
 
-Abstract: *Recently, 3D Gaussian splatting (3D-GS) has gained popularity in novel-view scene synthesis. It addresses the challenges of lengthy training times and slow rendering speeds associated with Neural Radiance Fields (NeRFs). Through rapid, differentiable rasterization of 3D Gaussians, 3D-GS achieves real-time rendering and accelerated training. They, however, demand substantial memory resources for both training and storage, as they require millions of Gaussians in their point cloud representation for each scene. We present a technique utilizing quantized embeddings to significantly reduce memory storage requirements and a coarse-to-fine training strategy for a faster and more stable optimization of the Gaussian point clouds. Our approach results in scene representations with fewer Gaussians and quantized representations, leading to faster training times and rendering speeds for real-time rendering of high resolution scenes. We reduce memory by more than an order of magnitude all while maintaining the reconstruction quality. We validate the effectiveness of our approach on a variety of datasets and scenes preserving the visual quality while consuming 10-20x less memory and faster training/inference speed.*
+Abstract: *Recently, 3D Gaussian splatting (3D-GS) has gained popularity in novel-view scene synthesis. It addresses the challenges of lengthy training times and slow rendering speeds associated with Neural Radiance Fields (NeRFs). Through rapid, differentiable rasterization of 3D Gaussians, 3D-GS achieves real-time rendering and accelerated training.  They, however, demand substantial memory resources for both training and storage, as they require millions of Gaussians in their point cloud representation for each scene. We present a technique utilizing quantized embeddings to significantly reduce per-point memory storage requirements and a coarse-to-fine training strategy for a faster and more stable optimization of the Gaussian point clouds. Our approach develops a pruning stage which results in scene representations with fewer Gaussians, leading to faster training times and rendering speeds for real-time rendering of high resolution scenes. We reduce storage memory by more than an order of magnitude all while preserving the reconstruction quality. We validate the effectiveness of our approach on a variety of datasets and scenes preserving the visual quality while consuming 10-20x less memory and faster training/inference speed.*
 
 ## Cloning the Repository
 The codebase is built off of the codebase of 3D-Gaussian Splatting (3D-GS) [here](https://github.com/graphdeco-inria/gaussian-splatting) by Kerbl et. al. The setup instructions are similar to their codebase with small changes in the libraries.

arguments/__init__.py

@@ -197,6 +197,9 @@ def __init__(self, parser, config):
         self.densify_from_iter = 500
         self.densify_until_iter = 15_000
         self.densify_grad_threshold = 0.0002
+        self.infl_prune_interval = 3000
+        self.quantile_threshold = 0.05
+        self.prune_until_iter = 24_000
         self.accumulate_fraction = 0.0
         self.search_best_iters = 0
         self.resize_period = 0.0

configs/efficient-3dgs.yaml

@@ -30,5 +30,9 @@ opt_params:
   resize_scale: 0.3
   transform: 'downsample'
   opacity_reset_interval: 2500
-  densification_interval: 250
+  densification_interval: 125
+  densify_until_iter: 18000
+  infl_prune_interval: 5100
+  prune_until_iter: 27000
+  quantile_threshold: 0.15
   log_interval: 100

gaussian_renderer/__init__.py

@@ -11,11 +11,13 @@
 
 import torch
 import math
-from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer
+# from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer
+from diff_gaussian_rasterization_depth import GaussianRasterizationSettings, GaussianRasterizer
 from scene.gaussian_model import GaussianModel
 from utils.sh_utils import eval_sh
 
-def render(viewpoint_camera, pc : GaussianModel, pipe, bg_color : torch.Tensor, image_shape=None, scaling_modifier = 1.0, override_color = None):
+def render(viewpoint_camera, pc : GaussianModel, pipe, bg_color : torch.Tensor, image_shape=None, scaling_modifier = 1.0, 
+           override_color = None, render_depth = False, get_infl = False):
     """
     Render the scene. 
     
@@ -47,7 +49,9 @@ def render(viewpoint_camera, pc : GaussianModel, pipe, bg_color : torch.Tensor,
         sh_degree=pc.active_sh_degree,
         campos=viewpoint_camera.camera_center,
         prefiltered=False,
-        debug=pipe.debug
+        debug=pipe.debug,
+        render_depth=render_depth,
+        get_infl=get_infl
     )
 
     rasterizer = GaussianRasterizer(raster_settings=raster_settings)
@@ -84,7 +88,7 @@ def render(viewpoint_camera, pc : GaussianModel, pipe, bg_color : torch.Tensor,
         colors_precomp = override_color
 
     # Rasterize visible Gaussians to image, obtain their radii (on screen). 
-    rendered_image, radii = rasterizer(
+    rendered_image, infl, radii, depth = rasterizer(
         means3D = means3D,
         means2D = means2D,
         shs = shs,
@@ -99,4 +103,6 @@ def render(viewpoint_camera, pc : GaussianModel, pipe, bg_color : torch.Tensor,
     return {"render": rendered_image,
             "viewspace_points": screenspace_points,
             "visibility_filter" : radii > 0,
-            "radii": radii}
+            "radii": radii,
+            "depth": depth,
+            "influence": infl}

scene/gaussian_model_sq.py

@@ -36,6 +36,8 @@ def __init__(self, sh_degree : int, latent_args: QuantizeParams):
         self.max_radii2D = torch.empty(0)
         self.xyz_gradient_accum = torch.empty(0)
         self.denom = torch.empty(0)
+        self.influence = torch.empty(0)
+        self.infl_denom = torch.empty(0)
         self.optimizer = None
         self.percent_dense = 0
         self.spatial_lr_scale = 0
@@ -107,6 +109,8 @@ def capture(self):
             self.max_radii2D,
             self.xyz_gradient_accum,
             self.denom,
+            self.influence,
+            self.infl_denom,
             OrderedDict([(n, l.state_dict()) for n,l in self.latent_decoders.items()]),
             self.optimizer.state_dict(),
             self.spatial_lr_scale,
@@ -118,13 +122,17 @@ def restore(self, model_args, training_args):
         self.max_radii2D, 
         xyz_gradient_accum, 
         denom,
+        influence,
+        infl_denom,
         ldec_dicts,
         opt_dict, 
         self.spatial_lr_scale) = model_args
         self.training_setup(training_args)
         self.optimizer.load_state_dict(opt_dict)
         self.xyz_gradient_accum = xyz_gradient_accum
         self.denom = denom
+        self.influence = influence
+        self.infl_denom = infl_denom
         for n in ldec_dicts:
             self.latent_decoders[n].load_state_dict(ldec_dicts[n])
 
@@ -135,6 +143,8 @@ def capture_best_state(self):
             self.max_radii2D.detach().cpu(),
             self.xyz_gradient_accum.detach().cpu(),
             self.denom.detach().cpu(),
+            self.influence.detach().cpu(),
+            self.infl_denom.detach().cpu(),
             OrderedDict([(n, {k: v.detach().cpu() for k, v in l.state_dict().items()}) for n,l in self.latent_decoders.items()]),
             self.spatial_lr_scale,
         )
@@ -145,13 +155,17 @@ def restore_best_state(self, model_args, training_args):
         self.max_radii2D, 
         xyz_gradient_accum, 
         denom,
+        influence,
+        infl_denom,
         ldec_dicts,
         self.spatial_lr_scale) = model_args
         self.training_setup(training_args)
         self._latents = {k:v.cuda() for k,v in self._latents.items()}
         self.max_radii2D = self.max_radii2D.cuda()
         self.xyz_gradient_accum = xyz_gradient_accum.cuda()
         self.denom = denom.cuda()
+        self.influence = influence.cuda()
+        self.infl_denom = infl_denom.cuda()
         for n in ldec_dicts:
             self.latent_decoders[n].load_state_dict(ldec_dicts[n])
     @property
@@ -300,6 +314,8 @@ def training_setup(self, training_args):
         self.percent_dense = training_args.percent_dense
         self.xyz_gradient_accum = torch.zeros((self.get_xyz.shape[0], 1), device="cuda")
         self.denom = torch.zeros((self.get_xyz.shape[0], 1), device="cuda")
+        self.influence = torch.zeros((self.get_xyz.shape[0]), device="cuda")
+        self.infl_denom = torch.zeros((self.get_xyz.shape[0]), device="cuda")
 
         self.lr_scaling = OrderedDict()
         for i,param in enumerate(self.param_names):
@@ -500,6 +516,8 @@ def prune_points(self, mask):
 
         self.denom = self.denom[valid_points_mask]
         self.max_radii2D = self.max_radii2D[valid_points_mask]
+        self.influence = self.influence[valid_points_mask]
+        self.infl_denom = self.infl_denom[valid_points_mask]
 
     def cat_tensors_to_optimizer(self, tensors_dict):
         optimizable_tensors = {}
@@ -543,6 +561,9 @@ def densification_postfix(self, new_xyz, new_features_dc, new_features_rest, new
         self.denom = torch.zeros((self.get_xyz.shape[0], 1), device="cuda")
         self.max_radii2D = torch.zeros((self.get_xyz.shape[0]), device="cuda")
 
+        self.influence = torch.cat((self.influence,torch.zeros((new_xyz.shape[0]), device="cuda")))
+        self.infl_denom = torch.cat((self.infl_denom,torch.zeros((new_xyz.shape[0]), device="cuda")))
+
     def densify_and_split(self, grads, grad_threshold, scene_extent, N=2):
         n_init_points = self.get_xyz.shape[0]
         # Extract points that satisfy the gradient condition
@@ -630,4 +651,17 @@ def densify_and_prune(self, max_grad, min_opacity, extent, max_screen_size):
     def add_densification_stats(self, viewspace_point_tensor, update_filter):
         self.xyz_gradient_accum[update_filter] += torch.norm(viewspace_point_tensor.grad[update_filter,:2], dim=-1, keepdim=True)
         # self.xyz_gradient_accum[update_filter] += viewspace_point_tensor.grad[update_filter,:2]
-        self.denom[update_filter] += 1
+        self.denom[update_filter] += 1
+
+    @torch.no_grad()
+    def add_influence_stats(self, influence):
+        self.influence += influence
+        self.infl_denom[influence>0] += 1
+
+    @torch.no_grad()
+    def prune_influence(self, quantile_threshold):
+        threshold = torch.quantile(self.influence,quantile_threshold)
+        prune_mask = self.influence<=threshold
+        self.prune_points(prune_mask)
+        self.influence *= 0
+        self.infl_denom *= 0

submodules/diff-gaussian-rasterization-depth/.gitignore

@@ -0,0 +1,3 @@
+build/
+diff_gaussian_rasterization.egg-info/
+dist/

submodules/diff-gaussian-rasterization-depth/.gitmodules

@@ -0,0 +1,3 @@
+[submodule "third_party/glm"]
+	path = third_party/glm
+	url = https://github.com/g-truc/glm.git

submodules/diff-gaussian-rasterization-depth/CMakeLists.txt

@@ -0,0 +1,36 @@
+#
+# Copyright (C) 2023, Inria
+# GRAPHDECO research group, https://team.inria.fr/graphdeco
+# All rights reserved.
+#
+# This software is free for non-commercial, research and evaluation use 
+# under the terms of the LICENSE.md file.
+#
+# For inquiries contact  [email protected]
+#
+
+cmake_minimum_required(VERSION 3.20)
+
+project(DiffRast LANGUAGES CUDA CXX)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_EXTENSIONS OFF)
+set(CMAKE_CUDA_STANDARD 17)
+
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+
+add_library(CudaRasterizer
+	cuda_rasterizer/backward.h
+	cuda_rasterizer/backward.cu
+	cuda_rasterizer/forward.h
+	cuda_rasterizer/forward.cu
+	cuda_rasterizer/auxiliary.h
+	cuda_rasterizer/rasterizer_impl.cu
+	cuda_rasterizer/rasterizer_impl.h
+	cuda_rasterizer/rasterizer.h
+)
+
+set_target_properties(CudaRasterizer PROPERTIES CUDA_ARCHITECTURES "70;75;86")
+
+target_include_directories(CudaRasterizer PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/cuda_rasterizer)
+target_include_directories(CudaRasterizer PRIVATE third_party/glm ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})

submodules/diff-gaussian-rasterization-depth/LICENSE.md

@@ -0,0 +1,83 @@
+Gaussian-Splatting License  
+===========================  
+
+**Inria** and **the Max Planck Institut for Informatik (MPII)** hold all the ownership rights on the *Software* named **gaussian-splatting**.  
+The *Software* is in the process of being registered with the Agence pour la Protection des  
+Programmes (APP).  
+
+The *Software* is still being developed by the *Licensor*.  
+
+*Licensor*'s goal is to allow the research community to use, test and evaluate  
+the *Software*.  
+
+## 1.  Definitions  
+
+*Licensee* means any person or entity that uses the *Software* and distributes  
+its *Work*.  
+
+*Licensor* means the owners of the *Software*, i.e Inria and MPII  
+
+*Software* means the original work of authorship made available under this  
+License ie gaussian-splatting.  
+
+*Work* means the *Software* and any additions to or derivative works of the  
+*Software* that are made available under this License.  
+
+
+## 2.  Purpose  
+This license is intended to define the rights granted to the *Licensee* by  
+Licensors under the *Software*.  
+
+## 3.  Rights granted  
+
+For the above reasons Licensors have decided to distribute the *Software*.  
+Licensors grant non-exclusive rights to use the *Software* for research purposes  
+to research users (both academic and industrial), free of charge, without right  
+to sublicense.. The *Software* may be used "non-commercially", i.e., for research  
+and/or evaluation purposes only.  
+
+Subject to the terms and conditions of this License, you are granted a  
+non-exclusive, royalty-free, license to reproduce, prepare derivative works of,  
+publicly display, publicly perform and distribute its *Work* and any resulting  
+derivative works in any form.  
+
+## 4.  Limitations  
+
+**4.1 Redistribution.** You may reproduce or distribute the *Work* only if (a) you do  
+so under this License, (b) you include a complete copy of this License with  
+your distribution, and (c) you retain without modification any copyright,  
+patent, trademark, or attribution notices that are present in the *Work*.  
+
+**4.2 Derivative Works.** You may specify that additional or different terms apply  
+to the use, reproduction, and distribution of your derivative works of the *Work*  
+("Your Terms") only if (a) Your Terms provide that the use limitation in  
+Section 2 applies to your derivative works, and (b) you identify the specific  
+derivative works that are subject to Your Terms. Notwithstanding Your Terms,  
+this License (including the redistribution requirements in Section 3.1) will  
+continue to apply to the *Work* itself.  
+
+**4.3** Any other use without of prior consent of Licensors is prohibited. Research  
+users explicitly acknowledge having received from Licensors all information  
+allowing to appreciate the adequacy between of the *Software* and their needs and  
+to undertake all necessary precautions for its execution and use.  
+
+**4.4** The *Software* is provided both as a compiled library file and as source  
+code. In case of using the *Software* for a publication or other results obtained  
+through the use of the *Software*, users are strongly encouraged to cite the  
+corresponding publications as explained in the documentation of the *Software*.  
+
+## 5.  Disclaimer  
+
+THE USER CANNOT USE, EXPLOIT OR DISTRIBUTE THE *SOFTWARE* FOR COMMERCIAL PURPOSES  
+WITHOUT PRIOR AND EXPLICIT CONSENT OF LICENSORS. YOU MUST CONTACT INRIA FOR ANY  
+UNAUTHORIZED USE: [email protected] . ANY SUCH ACTION WILL  
+CONSTITUTE A FORGERY. THIS *SOFTWARE* IS PROVIDED "AS IS" WITHOUT ANY WARRANTIES  
+OF ANY NATURE AND ANY EXPRESS OR IMPLIED WARRANTIES, WITH REGARDS TO COMMERCIAL  
+USE, PROFESSIONNAL USE, LEGAL OR NOT, OR OTHER, OR COMMERCIALISATION OR  
+ADAPTATION. UNLESS EXPLICITLY PROVIDED BY LAW, IN NO EVENT, SHALL INRIA OR THE  
+AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR  
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE  
+GOODS OR SERVICES, LOSS OF USE, DATA, OR PROFITS OR BUSINESS INTERRUPTION)  
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT  
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING FROM, OUT OF OR  
+IN CONNECTION WITH THE *SOFTWARE* OR THE USE OR OTHER DEALINGS IN THE *SOFTWARE*.  

submodules/diff-gaussian-rasterization-depth/README.md

@@ -0,0 +1,19 @@
+# Differential Gaussian Rasterization
+
+Used as the rasterization engine for the paper "3D Gaussian Splatting for Real-Time Rendering of Radiance Fields". If you can make use of it in your own research, please be so kind to cite us.
+
+<section class="section" id="BibTeX">
+  <div class="container is-max-desktop content">
+    <h2 class="title">BibTeX</h2>
+    <pre><code>@Article{kerbl3Dgaussians,
+      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
+      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
+      journal      = {ACM Transactions on Graphics},
+      number       = {4},
+      volume       = {42},
+      month        = {July},
+      year         = {2023},
+      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
+}</code></pre>
+  </div>
+</section>