AI models that generate video do not work with pixel data directly. Raw video is simply too large and complex. Instead, these models operate on a compact, learned summary of the video. This compressed summary is called a video VAE latent. It is created by a special neural network component called a Variational Autoencoder (VAE) designed for video.

The VAE's job is to shrink a video in both space (its height and width) and time (its sequence of frames). This process, known as spatiotemporal compression, turns a bulky video file into a small, information-rich tensor that is manageable for generative models like those used in latent diffusion. Understanding this compression is key to understanding how AI video generation works.

Key takeaways

  • A video VAE latent is a compressed representation of a video in a learned feature space, not just a smaller, pixel-based version of the video.
  • Compression happens in both space (reducing height and width) and time (reducing the number of frames), a process called spatiotemporal compression (Source).
  • The level of compression creates a trade-off. Higher compression saves memory and compute but can lose details and motion fidelity, potentially causing blurriness or flicker (Source).
  • Video VAEs that process space and time together (3D VAEs) are better at capturing coherent motion than those that process each frame independently (2D VAEs).
  • The terms "latents" and "tokens" can mean different things. Latents are usually continuous vectors, while tokens are discrete values from a fixed codebook (Source).

How is video compressed for diffusion? A Step-by-Step Guide

A video VAE compresses a video by passing it through an encoder network, which shrinks its dimensions, and then trains a decoder network to reverse the process. A diffusion model is then trained to operate entirely within this compressed "latent space." This dramatically reduces the computational cost, as the model only sees the small latent representation, not the full-pixel video.

Here is a step-by-step breakdown of how the encoder creates a video VAE latent.

Step 1: Input video as a tensor

First, a video clip is represented as a single numerical block, or tensor. A tensor is a multi-dimensional array of numbers. For video, this tensor has four dimensions: time (the number of frames), height, width, and channels (typically 3 for Red, Green, and Blue colors).

  • Example Shape: [Time, Height, Width, Channels]

Step 2: Spatial compression

The VAE encoder applies a series of operations called convolutions to the tensor. These convolutions process the video frame by frame, or in small spatio-temporal blocks, to downsample the height and width. This is spatial compression. It works like creating a thumbnail, but instead of just shrinking pixels, it learns to preserve the most important visual features in a compact form.

Step 3: Temporal compression

Simultaneously, the encoder applies convolutions across the time dimension. This reduces the number of temporal "steps" in the latent representation (Source). Instead of having a latent representation for every single frame, the encoder might create one representation that summarizes the content and motion across a small group of frames. This is temporal compression.

Step 4: Latent tensor output

The final output of the encoder is the video VAE latent. This is a new, much smaller tensor. Its dimensions have changed significantly. The height, width, and time dimensions are all smaller. The channel dimension is different, too; instead of RGB color channels, it now has "latent channels," which are learned features that encode complex information about the video's content. These channels do not have a direct visual interpretation.

  • Example Shape: [Latent_Time, Latent_Height, Latent_Width, Latent_Channels]

Step 5: Decoding

The VAE also includes a decoder. During training, the decoder's job is to take the compressed latent tensor and reconstruct the original video as accurately as possible. The better the decoder can reconstruct the video, the more useful information the latent tensor contains. Once trained, this decoder is used at the very end of the generation process to turn a model's generated latent back into a viewable video.

Worked Example: A Tensor's Journey Through a Video VAE

To concretely illustrate the impact of compression ratios, let's walk through an example with specific numbers. These values are illustrative and inspired by configurations used in models like OD-VAE and the VAE for Wan2.1 (Source, Source). Note that for simplicity, this example applies uniform temporal compression, but some causal VAEs handle the first frame differently from subsequent frames.

1. Input Video Imagine we start with a short video clip with the following properties:

  • Frames (Time): 16
  • Resolution: 256x256 pixels
  • Color Channels: 3 (RGB)

The input tensor shape is [16, 256, 256, 3]. The total number of values in this tensor is: 16 frames × 256 height × 256 width × 3 channels = 3,145,728 values

2. Compression Factors Our VAE is configured to compress the video with the following factors:

  • Spatial Compression: 8x (reduces height and width by a factor of 8)
  • Temporal Compression: 4x (reduces the number of time steps by a factor of 4)
  • Latent Channels: 16 (the encoder will output a 16-channel feature representation)

3. Calculating the Latent Shape Now we apply these factors to our input dimensions:

  • Latent Time: 16 frames / 4 = 4
  • Latent Height: 256 pixels / 8 = 32
  • Latent Width: 256 pixels / 8 = 32
  • Latent Channels: 16 (this is a design choice, not a calculation from the input)

The final latent tensor shape is [4, 32, 32, 16].

4. Final Latent Size The total number of values in this new latent tensor is: 4 time_steps × 32 height × 32 width × 16 channels = 65,536 values

By passing the video through the VAE, we reduced the number of values from over 3 million to just over 65,000. This is a reduction of approximately 98%. The diffusion model only ever has to process this tiny latent tensor, which makes training and generating video vastly more efficient. Other efficiency metrics exist; for instance, the HiTVideo tokenizer reported approximately 70% lower bits per pixel than baseline tokenizers in an 8-second video experiment (Source).

What’s the Difference? 2D vs. 3D VAEs and Tokens vs. Latents

Not all video compression methods are the same. The architecture of the VAE and the nature of its output have significant effects on performance. Two common points of confusion are the difference between 2D and 3D VAEs and between continuous latents and discrete tokens.

A 2D VAE treats a video as a sequence of independent images, compressing each one separately. A 3D VAE uses convolutions that operate on a video's height, width, and time dimensions together, allowing it to understand and compress motion (Source).

Continuous latents are vectors of real numbers, which is the standard output for VAEs used in latent diffusion. Discrete tokens are integers that point to an entry in a learned "codebook," much like words in a vocabulary. This process, called quantization, is central to models that borrow ideas from LLM tokenization. Some designs, like CV-VAE, explicitly aim to make their video latent space compatible with existing image VAE latents, allowing video models to reuse components from pretrained image models (Source).

Here is a comparison of these approaches:

Approach How It Works Best For Potential Downsides
2D Frame-wise VAE Applies an image VAE to each video frame independently. Simplicity; leveraging pre-trained image VAEs. Can create flicker and temporal inconsistencies because it does not model motion between frames.
3D Spatiotemporal VAE Uses 3D convolutions to compress blocks of video across space and time together. Capturing smooth motion and creating temporally coherent video. More complex to design and train than a 2D VAE.
Continuous Latents The VAE encoder outputs a tensor of floating-point numbers. Standard latent diffusion models, as they can operate on a continuous space. The meaning of latent values can be less interpretable.
Discrete Tokens (VQ-VAE) A Vector Quantized VAE maps continuous features to the closest entry in a learned codebook. Advanced schemes can use hierarchical tokenization for different levels of detail (Source). Transformer-based autoregressive models that predict one token at a time. Quantization adds a layer of information loss; can require a large codebook.

How Does a VAE Affect Video Quality and Motion?

The choices made in designing a video VAE directly impact the final generated video. Because the VAE performs lossy compression, it permanently discards some information. The diffusion model cannot reliably recover details that the VAE encoder discarded.

Lossy Compression and Detail

High spatial compression is a trade-off between efficiency and fidelity. To achieve a small latent size, the VAE encoder might discard fine textures, small objects, or other high-frequency details. When the decoder reconstructs the video, these details are gone, which can result in a blurry or over-smoothed appearance (Source).

Temporal Compression and Motion

Similarly, high temporal compression affects how motion is represented. If the VAE has too few temporal steps in its latent, it may struggle to model fast or complex movements accurately. This can lead to artifacts like motion blur, where moving objects appear smeared, or lagging, where an object's movement feels sluggish and disconnected from the rest of the scene (Source).

Flicker and Inconsistency

One of the most common artifacts in AI-generated video is flicker. This often happens when the VAE's latent representation is not stable over time. If the latent vectors for consecutive frames change erratically, the decoder will reconstruct frames that have subtle, random differences in brightness, color, or texture. This lack of temporal consistency in video is perceived as flicker. A good 3D VAE is designed specifically to produce a smooth, consistent latent representation to minimize this effect.

The Reconstruction vs. Generation Gap

It is a common mistake to assume that a VAE that perfectly reconstructs its input video will automatically lead to great video generation. While good reconstruction is necessary, it is not sufficient. The latent space itself must be "well-behaved"—smooth and structured in a way that the diffusion model can easily learn. A VAE that creates a chaotic or disjointed latent space may reconstruct well but will be very difficult for a generative model to work with, leading to poor quality generations (Source).

Common Beginner Pitfalls

When working with video VAEs, it is easy to make incorrect assumptions. Here are a few common pitfalls to avoid:

  • Confusing latent channels with color channels: The channels in a latent tensor are learned features that encode complex information. They are not interpretable as RGB values.
  • Assuming all frames are treated equally: Causal VAEs often process the first frame differently from subsequent frames. This affects how you calculate latent dimensions and can impact streaming generation (Source).
  • Evaluating only on single-frame sharpness: A good VAE reconstruction needs more than sharp frames. It is crucial to also check for temporal issues like flicker, inconsistent object identity, and unnatural motion trajectories.
  • Mistaking reconstruction artifacts for generation errors: Since VAE compression is lossy, the decoded output will have artifacts. It is important not to blame the diffusion model for flaws that were introduced by the VAE itself.

When Not to Use a Video VAE

While video VAEs are powerful, they are not always the right tool. Their primary purpose is to create a compressed space for another generative model, like a diffusion model, to operate in.

Here are situations where using a video VAE might be unnecessary or counterproductive:

  • For simple video editing tasks: If you are performing tasks like color correction, stabilization, or adding overlays, you should work directly with the pixel data. Compressing and decompressing with a VAE would degrade quality for no benefit.
  • When perfect, lossless reconstruction is required: VAEs are inherently lossy. For archival or professional video workflows where every pixel matters, using a VAE for compression is not appropriate. Standard video codecs (like H.264 or AV1) are designed for this purpose.
  • For extremely short or low-resolution clips: If a video is already very small (e.g., a few frames at 64x64 resolution), the computational overhead of running it through a VAE might outweigh the benefits of compression. It may be feasible to run a diffusion model directly in the pixel space.
  • For analytical tasks: If your goal is video understanding—like object tracking or action recognition—you would typically use a model designed for that task, which extracts features relevant to the analysis, not for reconstruction.

A video VAE is a specialized component for a specific kind of generative modeling. It solves the problem of making large videos manageable for models that learn to generate new content from scratch. Outside of this context, other tools are usually more appropriate.

What to learn next

Sources and further reading