Neural Radiance Fields (NeRF) and 3D Gaussian Splatting are two leading techniques for creating 3D scenes from a collection of 2D images. Both excel at novel view synthesis, the process of generating new images from viewpoints that were not in the original dataset. However, they work in fundamentally different ways. The debate over Gaussian Splatting vs NeRF comes down to a trade-off between rendering speed, training time, memory usage, and geometric fidelity.

This guide explains the core mechanisms of each method, compares their performance head-to-head, and provides a clear framework for choosing the right one for your project.

Key takeaways

  • Representation: NeRF uses a continuous representation, storing the scene in the weights of a neural network. 3D Gaussian Splatting uses an explicit representation, storing the scene as a large collection of discrete 3D primitives called Gaussians.
  • Rendering Speed: Gaussian Splatting is significantly faster at rendering, often achieving real-time frame rates. It uses a highly parallelizable projection and blending process similar to rasterization. NeRF is slower because it must sample many points along a ray for each pixel (Source).
  • Training Time: The original 3D Gaussian Splatting method trains substantially faster than the original NeRF.
  • Memory Usage: NeRF models can sometimes be more compact, while Gaussian Splatting scenes can be very large, consuming significant GPU memory to store data for millions of individual Gaussians (Source).
  • Best For: Splatting is the current standard for applications needing real-time walkthroughs and fast reconstructions. NeRF and its variants are still vital for research, sparse-view captures, and use cases that demand a continuous geometric representation or compact models.

What is a Neural Radiance Field (NeRF)?

A Neural Radiance Field (NeRF) represents a 3D scene using a small neural network to store a continuous volumetric function. This function acts like a universal lookup table for the scene. It takes a 5D coordinate as input—a 3D location (x, y, z) and a 2D viewing direction (θ, φ)—and outputs the color (RGB) and volume density (σ) at that exact point (Source).

Think of the neural network as having "learned" what exists at every single point in 3D space.

To render an image from a specific viewpoint, NeRF uses a process based on classical volume rendering:

  1. For each pixel in the final image, a ray is cast from the camera into the scene.
  2. The method samples dozens or hundreds of points along this ray.
  3. At each sample point, it queries the neural network with the point's location and the ray's direction to get a color and density.
  4. These color and density values are mathematically combined along the ray to calculate the final color of the pixel. A high density value at a point means it will contribute more color and block light from points behind it.

The ability to input the viewing direction allows NeRF to model view-dependent effects like specular highlights on a shiny surface, which change as you move your head (Source). Training involves optimizing the network's weights until the rendered images closely match the original input photos.

What is 3D Gaussian Splatting?

3D Gaussian Splatting (3DGS) represents a scene using a large collection of explicit 3D primitives called Gaussians. Instead of a continuous function, the scene is composed of millions of these colorful, semi-transparent, and stretchable blobs. Each Gaussian is a simple geometric object with its own set of properties that are optimized during training.

A helpful analogy is thinking of a point cloud, but where each "point" has size, shape, color, and transparency.

The key properties of each 3D Gaussian are:

  • Position: Its XYZ location in 3D space.
  • Covariance: A property that defines its 3D shape and orientation (e.g., a flat disc, a long needle, or a sphere).
  • Color: Stored using Spherical Harmonics, which allows the color to change based on viewing direction.
  • Opacity (Alpha): How transparent or opaque it is.

Rendering is achieved by "splatting" these 3D Gaussians onto the 2D image plane (Source). This process is more like traditional video game rasterization than NeRF's volume rendering. For each frame, all Gaussians are projected into 2D shapes, sorted from back to front, and blended together to form the final pixel colors. This is extremely fast because it avoids querying a slow neural network for every sample point and instead uses a highly optimized GPU-based sorting and blending pipeline.

How They Work: A Single Ray's Journey

The fundamental difference between Gaussian Splatting vs NeRF becomes clear when we follow the path of a single camera ray, which determines the color of one pixel.

Illustrative Example: Rendering One Pixel

  1. Camera Ray: Both methods begin identically. A ray is cast from the virtual camera's position, passing through the center of a pixel on the image plane and extending into the 3D scene.

  2. NeRF: Sampling the Field

    • The ray marches through the continuous field represented by the neural network.
    • It stops at many predefined sample points along its path.
    • At each of these points, it calls the neural network. The network takes the point's (x, y, z) coordinate and the ray's direction (θ, φ) as input.
    • The network outputs a (color, density) value for each of the points.
    • Finally, a volume rendering formula integrates these color/density pairs, weighted by density, to compute the single final color for the pixel.
  3. Gaussian Splatting: Projecting Primitives

    • The ray doesn't sample points along its path.
    • Instead, each 3D Gaussian in the scene is projected onto the 2D image plane, creating a "splat" or footprint.
    • These projected splats are sorted by depth.
    • The pixel's final color is calculated by alpha-compositing all the splats that cover it, starting from the one furthest back and blending forward. This avoids per-ray network queries entirely.

This difference—sampling a neural function vs. projecting explicit primitives—is the primary reason for the vast speed difference during rendering.

Gaussian Splatting vs NeRF: A Head-to-Head Comparison

While both achieve photorealistic results, their underlying architectures lead to different strengths and weaknesses.

Feature Neural Radiance Field (NeRF) 3D Gaussian Splatting
Core Representation Implicit & Continuous: A neural network (MLP) learns a function f(x, y, z, θ, φ) -> (RGB, σ). Explicit & Discrete: A collection of millions of 3D Gaussians, each with explicit properties (position, shape, color, opacity).
Rendering Process Volume Rendering: Samples many points along each ray and queries the network at each point. Computationally intensive. Rasterization: Projects 3D Gaussians to 2D splats, then sorts and alpha-blends them. Very fast and GPU-friendly.
Rendering Speed Slow (often <1 FPS for original NeRF). Variants have improved this but rarely match splatting. Very Fast (often >60 FPS), enabling real-time walkthroughs and interaction.
Training Speed Slow (original NeRF takes many hours to days). Fast (often 5-45 minutes).
Memory Usage Model can be compact (MBs), but training and acceleration structures can use significant VRAM (Source). Can be very high (GBs), as it stores data for every single Gaussian. A major bottleneck for large, detailed scenes.
Geometric Quality Can produce smoother, more coherent surfaces. Density is defined everywhere. Can suffer from "floaters" or noisy geometry, even when images look perfect. Geometry is composed of discrete blobs (Source).

When Should You Choose NeRF over Gaussian Splatting?

Despite the dominance of Gaussian Splatting in speed, there are important scenarios where NeRF-based methods are a better choice. The idea that splatting has made NeRF obsolete is a misconception (Source).

Choose a NeRF-based approach when:

  1. Model Compactness is Critical: If your final goal is a small file size for distribution, a compressed NeRF model is often much smaller than a file containing millions of Gaussians.
  2. You Need a True Continuous Representation: For applications in science or engineering that require querying the density or color at any arbitrary point in space (e.g., for simulations), NeRF's continuous function is superior to a discrete set of Gaussians.
  3. Input Views are Very Sparse: Some NeRF variants are designed to handle reconstructions from very few input images more gracefully than Gaussian Splatting, which may struggle to fill in large gaps.
  4. You Need to Handle Unbounded, 360° Scenes: Advanced NeRF models like Mip-NeRF 360 are specifically designed to handle large-scale, "unbounded" scenes without the aliasing and scaling issues that can affect original splatting methods (Source).
  5. Geometric Smoothness is a Priority: When extracting a 3D mesh from the representation, NeRF's underlying continuous density field can sometimes produce smoother and more complete surfaces than those derived from a cloud of Gaussians.

Common Mistakes to Avoid

Whether you are a beginner or an intermediate user, it's easy to fall into common traps when working with these technologies.

  • Mistaking Photorealism for Geometric Accuracy: This is the most common error. A beautifully rendered image from either method does not guarantee the underlying 3D geometry is correct. Both can produce "floaters" (bits of color floating in empty space) or misrepresent thin structures. Always verify the geometry with depth map visualizations if accuracy is important (Source).
  • Ignoring Camera Pose Quality: Both NeRF and Gaussian Splatting rely on accurate camera position and orientation data for each input image. If this data (usually generated by a Structure-from-Motion process) is poor, the final result will be blurry or distorted.
  • Expecting Good Results Far Outside the Capture Area: These methods are best at interpolating new views within the area covered by the original cameras. Extrapolating to a viewpoint far away will likely result in significant artifacts (Source).
  • Overlooking Aliasing: The original Gaussian Splatting can suffer from aliasing (shimmering or jagged artifacts) when viewing scenes from far away or at different resolutions. Newer methods like Mip-Splatting are designed to fix this, but it's something to watch for (Source).

Ultimately, neither NeRF nor Gaussian Splatting is a magic bullet for 3D reconstruction AI. They are powerful tools for novel view synthesis, and choosing between them requires understanding their core trade-offs in speed, memory, and the nature of their underlying representations.

What to learn next

Sources and further reading