[ratedgene]

I love the gaussian splatting that's going on. I also love the people pushing gaussian splitting and generative AI. I really feel there is something there but I'm not quite sure what yet. It's cool seeing this unfold, but I'm also worried it can turn into something like Photosynth, where it was a cool exercise but not much came out of it. I would love someone's input who is involved in this tech that could blue sky where it could be applied in interesting ways.

[andybak]

> like Photosynth, where it was a cool exercise but not much came out of it

It's hard to know what Photosynth could have become. It was shut down by Microsoft for unknown reasons. If it was open source it might have evolved and might still be around.

Gaussian Splatting has multiple implementations on both the training/generation side and the rendering side.

[mackopes]

100%. I really think Gaussian Splatting as a method can offer a lot, but it's also not powerful enough (yet) to be useful. Specifically editing and relighting GS models is rather clunky.

I think SplatGallery is an exploration of how people use the technology and what can it be useful for today and what is the next step.

[neomantra]

Gaussian splatting has been my obsession lately =)

Hard to believe, but the main technique is just over a year old (built on the shoulder of giants). This is the seminal paper for it [1] and here's a three-hour video so you really understand it [2] (and great how-to-really-read-a-paper-if-you-are-serious video). So what's great is a bunch of labs saw that and started building on it, so in the last three months there have been so many great improvements. And so much work done openly!

This has a great video roll [3] of some recent work, including use in construction and forestry.

If you have an Apple Vision, try MetalSplatter [4] and you will get an idea of how OMFG this stuff is.

We are in such a rich time of new compression knowledge and reality into weird representations!

I've been trying to evangelize it, but it takes so much foundation to understand how interesting it is. Many people (even software engineers and computer scientists) don't understand traditional 3D rendering pipelines and meshes/triangles and lighting. So you have to explain that, then concepts of spherical harmonics, gaussians with affine transforms, and the miracle that happens when you sample millions of them using raycasting at 800 fps. The neural network approaches go at 2fps.

In ideating and communicating the possibilities, I try to focus on the workflows...

Capture: we can sample lighting and depth of scenes with our phones or fancy cameras or drones; we can also use photogrammetry (algos/math to create depth fields from photos). This isn't specific to 3DGS, but 3DGS empowers it to be useful. So we have this tech where we can more easily capture objects and environments, edit them, and play it back. 3D captures has been around a long time (e.g. I worked on the Immersion Microscribe in the mid0-90s [4] and this is what machine vision used to be about), but we didn't have techniques to infer structure from the point clouds.

Processing: magic math turns this into a bunch of Gaussians with a transformation matrix (bell curves of different shape floating around) which represent the structure. Literally the components are spherical harmonics (color), density (alpha), variance, translation, rotation. Scenes will have many hundreds of thousands to tens of millions of them.

I tend to mention how LLMs capture aspects of knowledge into a big sea of weights that get computation applied to them and that this is similar very abstractly; and researchers have worked with Neural Radiance Fields. But what's great about Gaussian+Transform is that this is you can actually get an intuition of what's going on -- and the editors let you edit with the gaussians and filter/prune them. You can't do that with an NN (have intuition and direct manipulation).

Rendering: those objects are sampled and drawn on at interactive rates. You can represent scenes or objects. You can commingle them with traditional 3D assets. Works on recents phones very well. So this tech is broadly available now for playback.

The thing about it is that they lack structure. It's can be very ghostly and ethereal. So I think in the near term, this tech will be fantastic for customization/personal object capture/integration with scenes... but not for simulation but for human communication. As noted, the forestry and construction videos hint at this. Also product displays in website -- here's a Shopify plugin [6].

I have so much to say about it, but will stop here =)

[1] https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

[2] https://www.youtube.com/watch?v=xgwvU7S0K-k&t=1s

[3] https://www.youtube.com/playlist?list=PLrhy9mGYkm0aZnjL-4OpO...

[4] https://apps.apple.com/us/app/metalsplatter/id6476895334

[5] https://revware.net/microscribe-portable-cmm/microscribe-i-p...

[6] https://bitbybit.dev