A diffusion transformer (DiT) is a neural network architecture that uses a Transformer to denoise images within a diffusion model. The original diffusion transformer DiT paper by William Peebles and Saining Xie, published December 19, 2022, introduced this approach (Scalable Diffusion Models with Transformers). It treats an image as a sequence of patches, applying the scaling properties of Transformers to image generation. This represents a shift from the convolutional U-Net architectures common in earlier diffusion models.
Key takeaways
- A Diffusion Transformer (DiT) is a denoiser backbone for a diffusion model, replacing the traditional U-Net architecture with a Transformer. An official PyTorch implementation is provided with the original work.
- DiTs operate on sequences of 'patch tokens' derived from a compressed latent image, not on raw pixels, making computation more manageable.
- The architecture's main advantage is its proven scalability; increasing the model's size or the number of tokens consistently improves performance, achieving an FID score of 2.27 on ImageNet in the original paper.
- A DiT is only one component of a full generative system, which also requires a VAE (autoencoder), a noise scheduler, a sampler, and a conditioning mechanism.
- DiT-style architectures can be applied to domains beyond image generation, with published work on tasks like text-to-speech.
What is a Diffusion Transformer (DiT)?
A Diffusion Transformer (DiT) is a type of denoising network that powers a diffusion model. Instead of using the common U-Net design, a DiT uses a Transformer architecture to predict and remove noise from a noisy image representation. It treats a compressed image as a sequence of smaller pieces called patches, much like a language model treats a paragraph as a sequence of words. This allows it to learn relationships between different parts of the image.
The DiT itself is not a complete image generation system. It is the core engine inside a larger pipeline. The full system needs other parts to work: an autoencoder to compress the image, a scheduler to control the noise level, and a sampler to guide the generation process step-by-step. The DiT's specific job is, at each step, to look at a noisy image and predict the noise that needs to be removed (Hugging Face documentation).
For a refresher on the full diffusion process, see our guide on how diffusion models work.
How Does the DiT Architecture Work? (Step-by-Step)
The Diffusion Transformer (DiT) architecture acts as the denoiser within a single step of the diffusion process. It takes a noisy latent image, a timestep, and optional conditioning as input, and outputs a prediction of the noise. This process is repeated multiple times to generate a final image. Here is how one of those steps works inside a latent DiT system.
Image to Latent Space: A high-resolution image is first compressed by a Variational Autoencoder (VAE) into a much smaller latent representation. For example, a 256x256 pixel image might become a 32x32 latent grid. This step makes the entire process more computationally efficient. You can read more about this in our guide to latent diffusion models.
"Patchify" the Latent Image: The noisy 32x32 latent grid is divided into non-overlapping patches. For instance, using a patch size of 2x2 would create a grid of 16x16 patches. Each of these small patches is then flattened into a single vector. This transforms the spatial grid into a sequence of 256 vectors, which are called patch tokens.
Add Positional and Conditioning Info: The model needs to know where each patch came from. Positional embeddings are added to each patch token to encode its original location in the grid. Separately, the current noise level (timestep) and any class or text information (conditioning) are converted into their own embedding vectors.
Process with Transformer Blocks: The sequence of patch tokens is fed through a series of standard Transformer blocks. Inside each block, self-attention allows every patch token to exchange information with every other patch token. The timestep and conditioning embeddings are injected into each block to guide the denoising, often using a method called adaptive layer normalization (adaLN).
Predict the Output: After passing through the final Transformer block, the output tokens are processed by a final linear layer. This layer reshapes the token sequence back into a grid with the same dimensions as the latent image (e.g., 32x32). This final grid represents the model's prediction of the noise that was added to the latent image at the current timestep.
A Worked Example: From Image to Tokens and Cost
The number of patch tokens is a critical factor in a DiT's performance and computational cost. Using smaller patches creates more tokens, which can capture more detail but dramatically increases the computation required by the self-attention mechanism. The cost of self-attention scales quadratically with the number of tokens. Let's walk through an example.
Scenario: We start with a 256x256 pixel image.
- Step 1: VAE Encoding. The VAE's encoder compresses the 256x256 image into a 32x32 latent grid. Let's assume it has 4 channels, so its shape is
32x32x4.
Now, we will see how patch size affects the token count and computational load.
Case 1: Patch Size = 4
- Step 2: Patchify. We divide the 32x32 latent grid into non-overlapping patches of size 4x4.
- Token Calculation: The number of patches along each dimension is
32 / 4 = 8. - Total Tokens: The total number of tokens is
8 * 8 = 64. - Computational Cost: The self-attention mechanism's cost is proportional to the square of the sequence length, so it's roughly proportional to
64² = 4,096.
Case 2: Patch Size = 2 (like DiT-XL/2)
- Step 2: Patchify. We divide the same 32x32 latent grid into smaller patches of size 2x2.
- Token Calculation: The number of patches along each dimension is
32 / 2 = 16. - Total Tokens: The total number of tokens is
16 * 16 = 256. This is four times as many tokens as in Case 1. - Computational Cost: The self-attention cost is now proportional to
256² = 65,536.
By halving the patch size from 4 to 2, we increased the number of tokens by 4x (from 64 to 256). This, in turn, increased the computational cost of the attention mechanism by approximately 16x (65,536 / 4,096). This trade-off is central to DiT design: smaller patches allow the model to see finer details but come at a significant computational price. The DiT-XL/2 model used this smaller patch size to achieve its strong benchmark results (see the DiT repository).
Why do transformers scale better than U-Nets here?
Transformers scale better than U-Nets in this context because increasing their size leads to more predictable and consistent improvements in generation quality. This "scaling law" behavior, first observed in large language models, means that investing more compute—by making the Transformer deeper, wider, or processing more tokens—reliably lowers the Fréchet Inception Distance (FID). The original paper demonstrated this by achieving a state-of-the-art FID of 2.27 on class-conditional ImageNet generation (Scalable Diffusion Models with Transformers).
While U-Nets can also be scaled, their complex, hierarchical structure with skip connections makes the effect of adding layers less direct. In contrast, a Transformer is a more uniform stack of identical blocks, making its scaling behavior simpler to study and predict. The global self-attention in Transformers, which allows any two image patches to interact directly, may also be a more efficient way to model complex, long-range dependencies than the localized operations of convolutions in a U-Net.
| Feature | Diffusion U-Net | Diffusion Transformer (DiT) |
|---|---|---|
| Core Operation | Convolutional layers | Self-attention mechanism |
| Data Structure | Multi-scale feature maps | A flat sequence of patch tokens |
| Information Flow | Hierarchical (downsample then upsample) with skip connections | Global; every token can attend to every other token in one block |
| Inductive Bias | Strong locality and translation equivariance from convolutions | Weaker spatial bias; relationships are learned via self-attention |
| Scaling Behavior | Performance improves with size, but the relationship is complex | Performance improves predictably with depth, width, or token count |
How is conditioning handled with adaLN-Zero?
A denoiser needs to be controlled by two key pieces of information: the current noise level (timestep) and the content to be generated (a class label or text prompt). DiT uses a technique called Adaptive Layer Normalization (adaLN) to inject this information directly into each Transformer block. The 'Zero' part is an initialization trick that helps stabilize training for very deep models.
Instead of simply adding conditioning vectors to the input tokens, adaLN uses them to modulate the entire network's activations. Here's how it works:
- The timestep and class label are first converted into embedding vectors.
- These vectors are then passed through a small neural network to produce two new vectors: a scale (
γ) and a shift (β). - Inside each Transformer block, the standard Layer Normalization operation is modified. After normalizing the activations, they are multiplied by the custom scale
γand have the custom shiftβadded to them.
This allows the conditioning information to dynamically control the output of every Transformer block, effectively "steering" the denoising process.
The "Zero" in adaLN-Zero refers to initializing the final projection layer of each residual block to multiply its output by zero. This means at the start of training, every block initially acts as an identity function (passing its input through unchanged). This simple trick creates a clean "shortcut" for gradients to flow through the network, which helps stabilize training, especially in very deep Transformers.
What are the common mistakes and limits to know about DiTs?
Diffusion Transformers are powerful, but it's easy to misunderstand their role and limitations. Avoiding common pitfalls can help you better understand and use these models effectively.
- Mistake: Confusing the DiT with the full system. A DiT is just the denoiser. A DiT checkpoint is useless without the matching VAE for encoding/decoding and the correct sampler and noise schedule. The complete system is required for image generation (as the Hugging Face documentation explains).
- Mistake: Treating 'DiT' as a generic term. While other models, like Stable Diffusion 3, have adopted Transformer-based components, their designs are not identical to the original. 'DiT' refers to the specific architecture from the 2022 paper, not every diffusion model that uses a Transformer.
- Mistake: Attributing all performance differences to the backbone. When comparing models, it's incorrect to attribute every difference to the DiT vs. U-Net choice without accounting for variations in parameter counts, training budgets, token counts, samplers, or guidance settings.
- Mistake: Treating FID as a complete measure of quality. The FID score is a useful benchmark but does not fully capture image diversity, semantic alignment with a prompt, or overall human preference.
- Limit: The quadratic cost of self-attention. A DiT's computational cost grows with the square of the number of tokens. This makes it very expensive to generate very high-resolution images or use very small patch sizes without specialized attention mechanisms.
- Limit: GFLOPs don't equal wall-clock speed. The theoretical compute (GFLOPs) of a model does not directly predict its real-world inference speed. Factors like memory bandwidth, specific hardware, kernel implementations, and sequence length also have a major impact.
- Limit: Dependence on the VAE. Because DiTs operate in latent space, the final image quality is capped by the quality of the VAE. A blurry or artifact-prone VAE will produce poor images, no matter how powerful the DiT is.
- Limit: Self-attention is not a silver bullet. A Transformer architecture cannot compensate for fundamental issues like poor data quality, a weak conditioning signal, an unsuitable noise schedule, or inadequate training compute.
- Limit: The cost of iterative sampling. Even with an efficient denoiser model, the overall inference process remains costly because the model must be run repeatedly (often 20-50 times) to generate a single image.
Current State of Diffusion Transformers
Diffusion Transformers are an established alternative to U-Net backbones for denoising in diffusion models. The original 2022 paper remains an essential reference, demonstrating a clear empirical relationship between computational scale (GFLOPs) and performance on benchmarks like ImageNet FID. The main appeal of the architecture is its regularity and its compatibility with proven Transformer scaling practices. However, DiTs do not eliminate fundamental costs associated with generative modeling, such as the quadratic complexity of attention with token growth, the overhead of latent decoding, or the computational cost of iterative sampling. Following the original paper, DiT-style architectures have been adapted for applications beyond image generation, with published work applying them to tasks like text-to-speech (DiT-TTS).
What to learn next
- FID Score Explained: A Guide to Image Generation Metrics - Understand Fréchet Inception Distance (FID), how it's calculated, and its limitations for evaluating modern AI image generators. Learn to in
- Image Inpainting Diffusion: A Beginner's Technical Guide - Learn how image inpainting diffusion works. This guide explains the masked denoising process to help you remove objects and fix common editi
- Classifier-Free Guidance: Learn how DiTs and other conditional diffusion models are controlled during inference to improve adherence to prompts with Classifier-Free Guidance (CFG) Explained.
- Transformer Architecture: Dive deeper into the self-attention and block structure that powers DiTs with our detailed guide on the Transformer Architecture Explained.
- Diffusion Samplers: Understand how the output from the DiT is used step-by-step to generate an image in our overview of Diffusion Samplers Explained.
