[keyle]

It will be interesting to see the first AAA game that uses these methods instead of rendering a 3D world. Even if made from CGI worlds, it would be a very interesting approach and with somewhat predictable performances.

Reminds me of Ecstatica [1], a 1994 game that had intense visuals with a very odd/different rendering engine made of 3D ellipsoids; in a way really crude splats in gouraud shading.

[1] https://ecstatica.fandom.com/wiki/Ecstatica

[dagmx]

I know this comes up a lot on HN because its not primarily a graphics community but:

1. Gaussian Splats are very expensive to render. They capture a lot of detail which makes them seem cheaper than an equivalent raster render of that quality, but they wouldn't meet real time AAA game performance requirements

2. Gaussian Splats don't have a concrete surface. Want to cast shadows or do physics? It's doable but very tricky. Want to relight them? Also tricky. What is the exact surface point that you want to affect or sample for any particular operation? Deformations also become very difficult to do well.

3. Gaussian Splats are not sharp. You can get sharper with different kernel types or higher density of points, but your costs go up as well.

4. Gaussian splats are awful for any kind of path tracing. You can do it but you go back to the issues above. So mixing and matching traditional content with splats becomes a performance bottleneck.

I don't think you'll see a AAA game use splats for more than something like cinematics in the near term.

[Brusco_RF]

I'm working on the vision component of a drone racing stack. Could I use GS to render my living room as a digital playground to train my vision models in?

I know nothing about the technology but the alternative is creating a 3d model of my living room which is also outside my skill-set.

[KaiserPro]

> Could I use GS to render my living room as a digital playground to train my vision models in?

Yes, its what the autonomous car people are doing.

However you might want to do photogrammetry first (https://github.com/alicevision/meshroom opensource) as that produces a mesh which you can use to detect collisions easier. The downside is that transparent objects render really badly. but it is a lot faster to render

[namibj]

Ehhh if it's just for looking and you don't have anything lidar just go for splats they're way better behaved, mostly because they don't need to understand a concept of "surface" they just understand "splat with spherical harmonics of view-dependant color".

[KaiserPro]

true, for visual only stuff they do work really well.

[dagmx]

Splats would work and it’s what a lot of automation folks use. But the risk is that your splats aren’t tight to the surface and that can cause false positives.

For training you can do a hybrid geometry plus splats workflow. Have geometry that you can constantly raycast against and have as an input to your vision training or to get accurate depth buffers.

The workflow for splats and photogrammetry are very similar.

[cyber_kinetist]

Note that the first published work of rendering Gaussian Volumes was in this 1991 paper (https://articles.tomasparks.name/publications/Westover1991.p...) - so 3DGS is really a rehash of an old method from the 90s!

The contributions of 3DGS lie in how fast you can make them in modern GPU hardware (tiling + sorting with threads), and how to make the pipeline differentiable so that you can fit the Gaussian splats with photogrammetry data. Similar to the history of deep learning, it became technically feasible once the GPU hardware was powerful enough.

[sqrt_1]

There was this FPS demo recently https://playcanv.as/p/qxGSuzYq/

People have also converted some small sections of Unreal 5 demos into splats https://superspl.at/scene/692c4f91

Or perhaps use a real world scan - it was suggested this one would make an ideal setting for zombies https://superspl.at/scene/6359774f

[monkpit]

Can someone explain the unreal demo linked here - is the reflection in the street also using splatting, or is it something else?

[speps]

Yes splats do reflections

[cyber_kinetist]

Yes but in a weird way (at least on vanilla 3DGS) - it duplicates the splats on the mirror side.

Vanilla 3DGS cannot do any specular lighting or reflections - the color is basically baked in the splats. There's some active research going on to create richer Gaussian splats so we can do shading (or even ray tracing on it) - but haven't seen anyone using in production yet.

[monkpit]

Is that weird, though? I thought that was a pretty common way to do reflections, but I’m not that well-versed in the latest tech in this space.

[grumbel]

Many years ago there was a game called Casebook[1], a small little detective game where you investigated rooms for clues. But unlike similar FMV games where you jumped from point to point, it had photorealistic environments that could be smoothly walk around in, much like later lightfield or gaussian splatting experiments.

[1] https://www.youtube.com/watch?v=o-VAaC5BgVE

[cubefox]

Any idea on how they achieved this?

[Cieric]

I can't say I know how they actually did it, but taking a look at the trailer I can point out that it looks like the spaces are confined and your character is on rails. I'm mainly going off of the instant direction changes that don't appear to be 45 degrees off from the camera direction. Once it's constrained down to a single line/path you could do some wild things like cube mapping a video, where the position in the video is tied to the characters position. I can't say I know how they would take that video though, my best guess there is the scenes are constructed in 3d software, just it was to expensive for real time rendering.

[cubefox]

Cube mapping a video sounds plausible, this is commonly known as 360° video. Putting the camera on rails (though I don't really notice rails in this case) and tying the video playback speed to the speed of the rail movement has also been done in the past in some pre-rendered PlayStation games, though without cube mapping. But I think it's not pre-rendered in this case. It looks far too realistic for a game that is at least 17 years old. My best guess: they captured the 360 degree videos with a real camera (stabilized in some way) and edited the equipment out frame by frame.

[grumbel]

The image capture was done with a robotic camera rig from what I understand, they photographed 360° images of the room from all possible position. They restricted the camera movement to a plane, which is why the player height is fixed. I don't know what they did on the software side with all the image.

[cubefox]

Oh cool, so the camera wasn't just on 1D "rails", it was on a 2D plane. I never before heard of a game (pre-rendered or photographed) which did that. Impressive.

[selimthegrim]

This reminds me of Titanic: Adventure Out of Time

[modeless]

Dreams for PS4 used point splatting and has a very unique look as a result. The splats were created from distance fields instead of being scanned, so they don't look like modern gaussian splats. They have a painterly look instead. https://youtu.be/2ltgkcoQzow

[avaer]

This is "rendering a 3D world". It's basically the exact same techniques that traditional rendering uses, just with a different primitive that's not triangles. Everything else pretty much carries over.

If you mean the technique of splatting specifically, Dreams for PS4 [1] is prior art.

If you mean pre-rendering, there's Myst and games like the original FF7 for PS1.

[1] https://en.wikipedia.org/wiki/Dreams_(video_game)

[accrual]

It's not gaussian splatting, but Outcast (1999) has an interesting voxel-like rendering for the world surface. It has a pretty distinct feeling when walking around in the early areas, and a somewhat clunky but usable UI.

> The game does not actually model three-dimensional volumes of voxels. Instead, it models the ground as a surface, which may be seen as being made up of voxels. The ground is decorated with objects that are modeled using texture-mapped polygons. When Outcast was developed, the term "voxel engine", when applied to video games, commonly referred to a ray casting engine (for example the Voxel Space engine). On the engine technology page of the game's website, the landscape engine is also referred to as the "Voxels engine". The engine is purely software based; it does not rely on hardware-acceleration via a 3D graphics card.

https://en.wikipedia.org/wiki/Outcast_(video_game)

[jayd16]

Bladerunner: Revelations used a similar technique to bake down large CGI worlds with expensive lighting into something that ran on a Pixel 1 at VR specs.

Its honestly really very hard to work with this stuff because you ultimately need to be able to meshes inside these scenes triangle seas and you need to do it in a way that plausibly fits in the world. You can't have unlit characters walking around a baked lit scene and have them fit in. That's just from a visual design perspective.

You also always want to have bounce light from your dynamic things onto the baked scene and depending on the tech, you might not even be able to spatially place a dynamic thing and have it properly occlude what splats it needs to occlude.

As is, its a niche technology for games. That might change one day.

https://github.com/googlevr/seurat https://www.youtube.com/watch?v=Pf5Q3bvXj8E

[jamwise]

I think it's inevitable it goes there. Right now the level of detail and quality of games is limited by the console/PC hardware you're playing on. But with the splats they can render the whole game's world in a massive server farm at Hollywood Movie quality. I imagine there might be some balance of splat and traditional rendering technology since not all objects will lend themselves well, but this might be truly transformative.

[dagmx]

Why would you limit one to your local hardware and one to a cloud infrastructure?

Both can be done locally or on cloud? the comparison point becomes moot if you change the parameters that drastically

[jamwise]

There wouldn't be any cloud. Splats are still local, but all the lighting and texture are pre-rendered. The problem is they're not interactive, so they'd be good for a lot of the environment but your main character and other things that need to be interactive would need to use a different approach.