[lxe]

Absolute noob question that I'm having a hard time understading:

In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?

[GistNoesis]

There are two aspects in the difference between NeRF and Gaussian Splatting :

- The first aspect concern how they solve the light rendering equation :

NeRF has more potential for rendering physical quality but is slower.

NeRF use raycasting. Gaussian Splatting project and draw gaussians directly in screen space.

Each have various rendering artefacts. One distinction is in handling light reflections. When you use raycasting, you can bounce your ray on mirror surfaces. Where as gaussian splatting, like alice in wonderland creates a symmetric world on the other side of the mirror (and when the mirror surface is curved, it's hopeless).

Although many NeRF don't implement reflections as a simplification, they can handle them almost natively.

Alternatively, NeRF is a volumetric representation, whereas Gaussian Splatting has surfaces baked in : Gaussian Splats are rendered in order front to back. This mean that when you have two thin objects one behind the other, like the two sides of a book, Gaussian splatting will be able to render the front and hide the back whereas NeRF will merge front and back because volumetric element are transparent. (Though in NeRF with spherical harmonics the Radiance Field direction will allow to cull back from front based on the viewing angle).

- The second aspect of NeRF vs Gaussian Splatting, is the choice of representation :

NeRF usually use a neural network to store the scene in a compressed form. Whereas Gaussian Splatting is more explicit and uncompressed, the scene is represented in a sort of "point cloud" fashion. This mean that if your scene has potential for compression, like repetitive textures or objects, then the NeRF will make use of it and hallucinate what's missing. Whereas gaussian splat will show holes.

Of course like this article is about, you can hybridize them.

[xrd]

This is a beautiful explanation, thanks so much!

[choppaface]

NeRF does glass, fog, reflections, and some furs better than gsplat. Gsplat does normal surfaces as well or better than NeRF, and gsplat also provides explicit geometry (a point cloud). NeRF models the entire image rendering function while gsplats only model typical surfaces.

This work (and others e.g. https://creiser.github.io/binary_opacity_grid/ ) attempt to blend the raytracing aspect of NeRF with the explicit surface aspect of gsplats.

One key non-research problem is that gsplats can render on mobile devices / headsets using vanilla WebGL APIs. But approaches like this paper require CUDA (and also apparently a top-shelf desktop GPU). If Apple and others (mostly Apple has been sandbagging tho) provided better support for WebGPU or an alternative API then NeRF research would be dramatically more impactful versus gsplats. The popularity of gsplats is largely due to its accessibility.

[zlenyk]

It's just a completely different paradigm of rendering and it's not clear which one will be dominant in the future. Gaussian splats are usually dependent on initialisation from point cloud, which makes whole process much more compliacated.