Add 3D gaussian splats support #5658
Conversation
| pixelRatio: {type: 'number', default: 1}, | ||
| xrPixelRatio: {type: 'number', default: 0.5} |
There was a problem hiding this comment.
These two properties don't belong on the component. They should probably be made part of the renderer system. Though the pixelRatio is a bit tricky given that we default to window.devicePixelRatio, so maybe express it as a factor on top of the devicePixelRatio?
There was a problem hiding this comment.
yeah. don't love the API yet either
| if (data.xrPixelRatio > 0) { | ||
| sceneEl.renderer.xr.setFramebufferScaleFactor(this.data.xrPixelRatio); | ||
| } | ||
| }, |
There was a problem hiding this comment.
Update doesn't handle changes to cutoutEntity.
src/components/splat.js
Outdated
| bytesDownloaded += newChunk.length; | ||
| chunks.push(newChunk); | ||
|
|
||
| // Downloar progress stats. |
There was a problem hiding this comment.
Typo
| // Downloar progress stats. | |
| // Download progress stats. |
| try { | ||
| var dataReceived = await reader.read(); | ||
| if (dataReceived.done) { | ||
| console.log('Completed download.'); |
There was a problem hiding this comment.
I'd prefer no logging from components, but if we do it should be using debug.
There was a problem hiding this comment.
yeah. I'll remove. still WIP.
src/components/splat.js
Outdated
|
|
||
| initGL: async function initGL (numVertices) { | ||
| console.log('initGL', numVertices); | ||
| this.el.object3D.frustumCulled = false; |
There was a problem hiding this comment.
This doesn't seem necessary. The actual mesh has frustum culling disabled as well further down in this method.
src/components/splat.js
Outdated
| // Wait until texture is ready | ||
| while (true) { | ||
| var centerAndScaleTextureProperties = renderer.properties.get(this.centerAndScaleTexture); | ||
| var covAndColorTextureProperties = renderer.properties.get(this.covAndColorTexture); | ||
| if (centerAndScaleTextureProperties && centerAndScaleTextureProperties.__webglTexture && | ||
| covAndColorTextureProperties && centerAndScaleTextureProperties.__webglTexture) { | ||
| break; | ||
| } | ||
| await new Promise(resolve => setTimeout(resolve, 10)); | ||
| } |
There was a problem hiding this comment.
Instead of this polling it should just tell Three.js to initialize the textures: https://threejs.org/docs/index.html#api/en/renderers/WebGLRenderer.initTexture
.eslintrc.json
Outdated
| @@ -83,6 +83,13 @@ | |||
| "ecmaVersion": 6 | |||
| } | |||
| }, | |||
| { | |||
| /* This module uses ES8 async / await due to WebXR Anchor Module integration */ | |||
There was a problem hiding this comment.
Comment incorrectly refers to WebXR Anchor Module. That being said, I don't believe async/await is a hard requirement for splat.js and it can be rewritten in the A-Frame code style for consistency, avoiding this exception altogether.
|
My main concern with a I haven't really been following Gaussian splatting developments all that closely, but quadjr's aframe component is pretty old. How does it compare to implementations in PlayCanvas, 8thwall/Scaniverse or gsplat.js? It seems that most have switched to compressed and quantized formats, although it seems no clear standard file format has emerged.
Please include the original license at the start of this file.
Into the Scaniverse (https://www.intothescaniverse.com/) mostly runs at 72fps on a Quest 3, so that's probably something we should aim for as well. Obviously depends on the amount of splats amongst other factors, but a small object-centered example would probably work better. Could even make it grabbable. |
|
from my perspective its to early to add this in the main branch, better to external git, ther eis a lot of discussion going on to bring splats to gltf/glb ( with mesh opt compression) |
A-Frame role has always been to be at the forefront: We built on custom elements v1 before being standard. We also shipped WebVR, WebXR + extensions (hand tracking, layers, immersive navigation…) and gltf before becoming standard. Standards are better designed when based on real world usage and feedback. Should be practice consolidation not prescription. Standard for gs3d probably premature. A-Frame mission is to make 3D content creation accesible to anyone. I see exciting opportunities at the intersection with AI. I would love for A-Frame to participate and be top of mind of researchers and early adopters. |
|
It would be very exciting to see A-Frame adopt 3DGS, even as part of an extras/experimental package. And A-Frame implementation of @mkkellogg's Three.js 3DGS viewer might be a better path to keep up with active development efforts. |
Thanks! I'm excited too. Looked at mkkellog implementation but it's pretty complicated / advanced already. I went with https://github.com/quadjr/aframe-gaussian-splatting that is a very minimal port of the original (first?) WebGL implementation https://github.com/antimatter15/splat My goal is to first have a minimal and performant implementation that we can later use as a base for more complex stuff. I would love to later have an example of a nice and simple viewer with great controls on every platform. |
A good chunk of the complexity of mkkellogg/GaussianSplats3D seems to be to make it performant and to support a wide array of formats/splat types. Both things that I'd argue are essential for a I don't really see the benefit in re-inventing the wheel in this case. Rendering of Gaussian splats is very expensive, so unless/until performance is on-par, wouldn't user just create their own components or use a community made one that does wrap any of the more mature implementations out there? |
Wheel far from invented. Stuff evolving quick. mkkellogg performance is ~24fps on Quest 3 on the simple demo scene. Playcanvas ~17fps (similar to ours) on the same truck splat on this PR's example. Scenes from Into the Scaniverse are the only ones I've seen decently performing on Quest 3 (~72fps). Presumably cherry picked. I want to develop understanding of the problem space before taking on any dependency. A simple single file implementation like this PR is a good starting point before incorporating anything with a lot of decisions made for us that we don't fully understand. |
|
I limited the number of splats rendered at any given time by adding the follow statement in line 532 We can render at 90fps on Quest3 up to ~150k splats. |
On the training, compression, rendering and sorting side there's plenty to experiment with. But this is about making it a core component of A-Frame. From the perspective of the user the relevance is being able to add "splats" to their scene. People will expect to be able to use the output of tools like SuperSplat and Scaniverse and have it perform comparable as well. In other words, we'd end up implementing loading of compressed .PLY files, .SPZ files, supporting the spherical harmonics for view dependent lighting and implementing the sorting in WASM using SharedArrayBuffer, progressive loading, perhaps even 2DGS support. All things that have been done numerous times. For experimenting, the core isn't the right place IMO. And for providing a component for users, we'd end up implementing (and maintaining!) a lot of features. I agree with @koktavy that wrapping mkkellogg/GaussianSplats3D is a better path to stay up to date. We can always contribute upstream if we do find improvements.
Both of those render at full resolution. I'm not able to reproduce ~17fps using the component in this PR when setting the
Not sure if they really cherry pick them, as they do allow users to create their own and post them on the map. From what I gather they do a lot of pre-processing. One thing I have noticed is that the background is actually static despite the Gaussian Splat appearance. This presumably helps quite a bit as it saves on the number of splats as well as reduces overdraw caused by large splats in the background.
Splats don't have a fixed cost associated with them. They vary in size and spatial distribution. Overdraw really kills performance AFAICT, so ~150k splats representing a "full scene" or ~150k focussed on a single object will likely perform wildly different. |
|
Any example of mkkellog working at least 72fps on Quest? At this point broad support of formats, or spherical harmonics are not top priority. Just interested on compelling examples that perform well and are interesting to look at. |
Last big item for 1.7.0 release. Still work in progress. Based on https://github.com/quadjr/aframe-gaussian-splatting implementation
I adapted style, made a few minor tweaks and commented thoroughly. I still don't fully understand some of the details. Good opportunity to deepen understanding of 3D gaussian splats for those interested. Before merge we should thoroughly understand the code and optimize.
I recommend getting acquainted with the original paper: https://arxiv.org/pdf/2308.04079
In the code comments you'll see references to other relevant literature.
In VR on a Quest 3 when the sample splat is fully loaded (~1.1M splats rendered) I get ~14fps.
Quick overview: