[lopsotronic]

Ah the timeless joy of falling through the floor geometry.

Seriously though - it's breathtaking.

The first guy who figures out the bridge between splats and dynamism - animation, editing, responsiveness - is going to be one of the immortals of 3d design.

[mrandish]

If you don't mind a basic but tangential question... I understand that splats like these are essentially a separate domain from traditional polygonal 3D scenes in a tool like Blender, but is it currently possible to use splats like the strawberry as elements in a Blender scene if they don't animate or interact any more than the camera / lights moving and being occluded by foreground objects?

Remembering how we used to cheat by projecting hi-res texture maps onto simple geometry to save modeling background objects with the trade-off of constraints on size, viewing angle, shadows, etc. With the relative ease of capturing these splats at high-fidelity, can they be used in a scene as projection-map-like 'fakes' except with far fewer constraints?

[Intralexical]

They're basically light fields, so even though they don't represent any geometry, you could easily integrate them into the rendering pipeline at the path tracing/rasterization stage for Cycles/EEVEE.

I think the "right" way to do it without forking Blender would be to write an OSL shader. Though the way I personally would have done it would be to convert the splat into an angle-dependent Math Node tree for an Emission shader, probably automatically with a Python script.

Trouble is light fields don't block geometry, so they would effectively blend additively and you would see right through them like a hologram. Easy enough to fix though, with a dummy mesh to composite out the background. Or you could just render the splat separately outside Blender, and composite your scene on top of it.

I'm not sure why you would want to, though. The splats couldn't be lit by any of the lights in your scene, nor can they receive shadows cast by any objects in your scene, because they're precomputed light fields. Objects in your scene could be lit by the splats, but it's not immediately clear to me that it would be much more useful or better than just having photos on billboard planes.

I think there might be a misunderstanding that splats provide a way to compute realistic lighting. In reality, they're only a way to store and sample pre-computed or measured lighting AFAIK, hence the need for OP's big camera rig. So integrating them into scene lighting or skeletal animation doesn't really apply.

[cubefox]

There are many ways to represent 3D data, but animations really only work properly with polygon meshes (e.g. triangle surface meshes or volumetric tetrahedral meshes).

[lopsotronic]

But the mesh is itself an abstraction, you just need to build that bridge.

We've been leaning away from pure polygons for decades, anyway. Vertex skinning, SDFs, volumetrics, simulation, and a lot more.

The meshes in a From Software game are for exmple hilariously simple, most of the animation is force simulation to make the famous "frizzles" that they like.

[cubefox]

Vertex skinning is essential for animation and it only works with polygons.

[lopsotronic]

I'm not sure that's completely accurate? Vertex skinning isn't (necessarily) tied to polygons . . but to having points (or any parameterized features) that can be transformed by a weighted blend of matrices.

The "vertex" in "vertex skinning" is really just "a thing with a position that gets moved."

p' = Σ wᵢ · Mᵢ · p

It's just a position. Triangles can come along for the ride downstream, but they're not essential, which is one of the reasons it's so efficient for some stuff. Polygons are the optimal surface - but surfaces are often extraneous.

Take all this a few hefty grains of salt, I'm an amateur in the field. My 3d/CAD work is strictly in support of my enterprise stuff. And making wicked battlemaps for gaming VTTs, natch.

But I will stand by the overarching statement that polygons are in fact an abstraction, and bridging that abstraction with whatever is in splats would be wicked awesome.

[data-ottawa]

I don’t know. Maybe today, but tomorrow?

If you can sample points inside a volume, in theory you could do that with splat geometry. If someone figures out a way to pass in animation time to a sampler, sample along geometry/wireframe or something else, and keep it from overly twinkling it might change everything.

I’m hand waving all the complexity into “if done one figures out”, of course.

I just don’t see why this method can’t evolve in the way diffusion models have evolved (knowing very little of the geberative mechanics of splats).

[cubefox]

> If someone figures out a way to pass in animation time to a sampler, sample along geometry/wireframe or something else, and keep it from overly twinkling it might change everything.

Not sure that's what you mean, but there was recently a paper where they put meshless (e.g. voxel or SDF) geometry in an animated tetrahedral mesh "cage" and then animate the meshless model by animating the mesh cage:

https://diglib.eg.org/server/api/core/bitstreams/bd94e19b-98...

https://youtube.com/watch?v=6lKAvxV2mno

https://youtube.com/watch?v=3c3-ue-fd88

Though this currently isn't compatible with 3DGS if I understand the limitations section correctly.

> Finally, our method operates unordered, limiting its suitability for complex volumetric effects. However, a potential solution lies in sorting the generated intervals for proper blending. This enhancement could improve our approach’s compatibility with various meshless representations, such as radiance fields and volumetric lighting.

[Intralexical]

Since splats sample the light field after surface reflection, you can't do realtime shading with splats the way you can with raytracing and rasterization. I guess it could be animated like a holographic movie, but not like a video game and not like a 3D editor, because the light for all angles in all frames has to be precomputed.

[db48x]

It’s easy but a bit data intensive. Take two 3D splat images at different times, optimize them, then interpolate from the first to the second. Repeat at intervals. Now you have a video. A full moving subject is about 500Mbps, although it depends a lot on the quality of the source images that you make the 3D splats from and how detailed the output image is. Search for “4D gaussian splats” to find references.

[cubefox]

That's just animation like an animated GIF: a series of static frames. What's more interesting is animation in the sense of deformation, as in skeletal/skinned meshes. These deformations require minimal data and can be generated dynamically.

[db48x]

I can’t think of any reason why it would be impossible to implement skeletal animation of gaussian splats. You still have to create the splats from images though. Just have your subject put their right arm in, then take your images and generate the second splat. Now you can interpolate between them to move the right arm. At least in that one axis. Repeat for other joints. Have them put their whole self in and turn them selves about, etc. How hard could it be? https://www.youtube.com/watch?v=aIqCSpuDwBE

[thfuran]

You pretty much just need a representation that can be constructed reasonably and interpolated.