From cc9bc1f1747fd9a465786cf13fad46930e723430 Mon Sep 17 00:00:00 2001 From: gmeanti Date: Tue, 16 May 2023 22:31:40 +0200 Subject: [PATCH 1/2] Add documentation for kplanes add-on --- docs/nerfology/methods/index.md | 1 + docs/nerfology/methods/kplanes.md | 77 +++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 docs/nerfology/methods/kplanes.md diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 4c8cf98fa7..780abf76a3 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -34,6 +34,7 @@ The following methods are supported in nerfstudio: Nerfacto Tetra-NeRF TensoRF + K-Planes ``` (own_method_docs)= diff --git a/docs/nerfology/methods/kplanes.md b/docs/nerfology/methods/kplanes.md new file mode 100644 index 0000000000..9766bbf50f --- /dev/null +++ b/docs/nerfology/methods/kplanes.md @@ -0,0 +1,77 @@ +# K-Planes + +

K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

+ + +```{button-link} https://sarafridov.github.io/K-Planes/ +:color: primary +:outline: +Paper Website +``` + +```{button-link} https://github.com/sarafridov/K-Plane +:color: primary +:outline: +Official Code +``` + +```{button-link} https://github.com/Giodiro/kplanes_nerfstudio +:color: primary +:outline: +Nerfstudio add-on code +``` + + + +**A unified model for static, dynamic and variable appearance NeRFs.** + + +## Installation + +First, install nerfstudio and its dependencies. Then install the K-Planes add-on +``` +pip install git+https://github.com/giodiro/kplanes_nerfstudio +``` + +## Running K-Planes + +Currently, there are two default configurations provided which use the blender and DNeRF dataloaders. However, you can easily extend them to create a new configuration for different datasets. + +The default configurations provided are +| Method | Description | Scene type | +| ----------------- | ------------------------------------------------- | ------------------------------ | +| `kplanes` | Config. tuned for synthetic NeRF (blender) scenes | static, synthetic | +| `kplanes-dynamic` | Config. tuned for DNeRF dataset | dynamic (monocular), synthetic | + + +for training with these two configurations you should run +```bash +ns-train kplanes --data +``` +or +```bash +ns-train kplanes-dynamic --data +``` + +## Method + +![method overview](https://sarafridov.github.io/K-Planes/assets/intro_figure.jpg)
+K-planes represents a scene in k dimensions -- where k can be 3 for static 3-dimensional scenes or 4 for scenes which change in time -- +using k-choose-2 planes (or grids). After ray-sampling, the querying the field at a certain position consists in querying each plane (with interpolation), and combining the resulting features through multiplication. +This factorization of space and time keeps memory usage low, and is very flexible in the kinds of priors and regularizers that can be added.
+
+ +We support hybrid models with a small MLP (left) and fully explicit models (right) + + + +We also support decomposing a scene into its space and time components + From ded2059c94c03c856dc6d582269c88f048adf958 Mon Sep 17 00:00:00 2001 From: gmeanti Date: Wed, 17 May 2023 20:26:38 +0200 Subject: [PATCH 2/2] Fix all integration spots with nerfstudio --- docs/index.md | 1 + docs/nerfology/methods/index.md | 2 +- docs/nerfology/methods/kplanes.md | 28 +++++++++++++++++--------- nerfstudio/configs/external_methods.py | 15 ++++++++++++++ 4 files changed, 36 insertions(+), 10 deletions(-) diff --git a/docs/index.md b/docs/index.md index abddd1f826..e346500182 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,6 +112,7 @@ This documentation is organized into 3 parts: ### Third-party Methods - [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions +- [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index cd66d27a7f..07eccb4b0b 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -28,13 +28,13 @@ The following methods are supported in nerfstudio: :maxdepth: 1 Instant-NGP Instruct-NeRF2NeRF + K-Planes LERF Mip-NeRF NeRF Nerfacto Tetra-NeRF TensoRF - K-Planes ``` (own_method_docs)= diff --git a/docs/nerfology/methods/kplanes.md b/docs/nerfology/methods/kplanes.md index 9766bbf50f..65c6c10444 100644 --- a/docs/nerfology/methods/kplanes.md +++ b/docs/nerfology/methods/kplanes.md @@ -1,6 +1,6 @@ # K-Planes -

K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

+

Explicit Radiance Fields in Space, Time, and Appearance

```{button-link} https://sarafridov.github.io/K-Planes/ @@ -32,18 +32,18 @@ Nerfstudio add-on code First, install nerfstudio and its dependencies. Then install the K-Planes add-on ``` -pip install git+https://github.com/giodiro/kplanes_nerfstudio +pip install kplanes-nerfstudio ``` ## Running K-Planes -Currently, there are two default configurations provided which use the blender and DNeRF dataloaders. However, you can easily extend them to create a new configuration for different datasets. +There are two default configurations provided which use the blender and DNeRF dataloaders. However, you can easily extend them to create a new configuration for different datasets. The default configurations provided are -| Method | Description | Scene type | -| ----------------- | ------------------------------------------------- | ------------------------------ | -| `kplanes` | Config. tuned for synthetic NeRF (blender) scenes | static, synthetic | -| `kplanes-dynamic` | Config. tuned for DNeRF dataset | dynamic (monocular), synthetic | +| Method | Description | Scene type | Memory | +| ----------------- | -------------------------| ------------------------------ | ------ | +| `kplanes` | Tuned for blender scenes | static, synthetic | ~4GB | +| `kplanes-dynamic` | Tuned for DNeRF dataset | dynamic (monocular), synthetic | ~5GB | for training with these two configurations you should run @@ -55,6 +55,14 @@ or ns-train kplanes-dynamic --data ``` +:::{admonition} Note +:class: warning + +`kplanes` is set up to use blender data, (download it running `ns-download-data blender`), +`kplanes-dynamic` is set up to use DNeRF data, (download it running `ns-download-data dnerf`). +::: + + ## Method ![method overview](https://sarafridov.github.io/K-Planes/assets/intro_figure.jpg)
@@ -63,7 +71,8 @@ using k-choose-2 planes (or grids). After ray-sampling, the querying the field a This factorization of space and time keeps memory usage low, and is very flexible in the kinds of priors and regularizers that can be added.

-We support hybrid models with a small MLP (left) and fully explicit models (right) +We support hybrid models with a small MLP (left) and fully explicit models (right), through the `linear_decoder` [configuration key](https://github.com/Giodiro/kplanes_nerfstudio/blob/db4130605159dadaf180228be5d0335d2c4d21f9/kplanes/kplanes.py#L87) +
@@ -71,7 +80,8 @@ We support hybrid models with a small MLP (left) and fully explicit models (righ -We also support decomposing a scene into its space and time components +The model also supports decomposing a scene into its space and time components. For more information on how to do this see the [official code repo](https://github.com/sarafridov/K-Plane) +
diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index b15632cab0..9a0f7e774b 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -59,6 +59,21 @@ class ExternalMethod: ) ) +# K-Planes +external_methods.append( + ExternalMethod( + """[bold yellow]K-Planes[/bold yellow] +For more information visit https://docs.nerf.studio/en/latest/nerfology/methods/kplanes.html + +To enable K-Planes, you must install it first by running: + [grey]pip install kplanes-nerfstudio[/grey]""", + configurations=[ + ("kplanes", "K-Planes model tuned to static blender scenes"), + ("kplanes-dynamic", "K-Planes model tuned to dynamic DNeRF scenes"), + ], + pip_package="kplanes-nerfstudio", + ) +) # LERF external_methods.append(