Diffusion models are the technology behind many popular AI image generators. They can create detailed images from simple text prompts. But how do diffusion models work? The process involves systematically destroying training images with noise and then teaching a neural network how to reverse that process, step by step.

This article explains the complete diffusion model lifecycle. You will learn about the two core processes, what the model actually learns during training, and how it uses that knowledge to generate new and original samples from pure noise.

Key takeaways

  • Diffusion models generate data by reversing a noise-adding process. They start with random noise and gradually refine it into a coherent sample like an image.
  • The system has two main parts: a fixed forward process that adds noise to training data, and a learned reverse process that removes it.
  • During training, the model doesn't learn to reconstruct the image directly. Instead, it is trained to predict the specific noise that was added at a given step.
  • Generation is a multi-step process. The model iteratively predicts noise, subtracts a portion of it from the current image, and adds a small amount of new randomness to produce novel results.
  • Text prompts guide generation by influencing the model's noise prediction at each step, steering the process toward the desired content.

What is a diffusion model?

A diffusion model is a type of generative model, which is a class of AI models designed to create new data. It works by learning to reverse a process of gradual corruption. Imagine taking a clear photograph and slowly adding random noise to it over hundreds of steps until it's an unrecognizable field of static. A diffusion model learns how to undo that process, starting from static and carefully removing noise step-by-step to arrive at a clear, new photograph.

Think of a sculptor who starts with a rough block of marble (random noise). The sculptor knows a set of rules for how to chip away at the marble (denoising) to reveal a statue. At each stage, they look at the current state of the block and decide where to make the next small chip. A diffusion model is like that set of rules, applied iteratively to refine noise into a structured output. This iterative refinement is a key difference from other generative models that might try to create an image in a single shot.

While known for images, this technique can also generate audio, video, and 3D data. The core concept involves a forward process of adding noise and a learned reverse process of removing it (Generative Diffusion Models: A Review and Taxonomy of Method, Advancements and Applications).

The Two Halves of a Diffusion Model

To understand how diffusion models work, it helps to split the system into its two fundamental parts: the forward process, which is fixed, and the reverse process, which is learned.

1. The Forward Process: Adding Noise

The forward process is a simple, fixed procedure that does not involve any learning. Its only purpose is to produce noisy images to train the model.

It starts with a clean image from a training dataset (let's call it x₀). Then, in a series of small steps, it adds a tiny amount of noise. This is repeated for a set number of timesteps, often denoted as T (e.g., 1000 steps). The noise is typically Gaussian noise, which is a type of statistical noise with a specific bell-curve distribution.

A "noise schedule" defines the amount of noise added at each timestep t. By the final timestep T, the original image is completely overwhelmed, and the result (x_T) is indistinguishable from pure random noise (Denoising Diffusion Probabilistic Models (NeurIPS 2020)).

A critical feature of this process is that we can calculate the state of the noisy image at any timestep t directly, without having to add noise step-by-step. This makes training very efficient.

2. The Reverse Process: Learning to Denoise

The reverse process is where the machine learning happens. Here, the goal is to train a neural network to undo one step of the forward process. The model is given a noisy image x_t from some timestep t and is trained to estimate the previous, slightly less noisy version, x_{t-1}.

If the model can learn to take this small step backward, it can repeat the process. By starting with pure noise (x_T) and applying the reverse step T times, it can generate a clean image (x₀). This learned reverse process is the heart of the diffusion model (Denoising Diffusion Probabilistic Models (NeurIPS 2020)).

The key insight from the paper that popularized this method, Denoising Diffusion Probabilistic Models (DDPMs), is that you don't need to predict the pixels of the previous image directly. Instead, you can train the model to predict the noise that was added between x_{t-1} and x_t (Denoising Diffusion Probabilistic Models). This simplifies the task and leads to more stable training.

What does the neural network actually predict?

The model's objective is simpler and more direct than predicting the final clean image. At any given step, the neural network is trained to predict the specific random noise that was used to corrupt the original image to its current state. This makes the learning task much more manageable.

Instead of asking the model, "What was the original image?" we ask, "What noise was added to the original image to get this noisy version?"

The training process works like this:

  1. Start with a clean image, x₀.
  2. Generate a random noise pattern, ε.
  3. Pick a random timestep, t.
  4. Use a direct formula to create the noisy image x_t by mixing x₀ and ε according to timestep t.
  5. Feed x_t and t to the neural network.
  6. The network predicts a noise pattern, ε_predicted.
  7. The model's error is the difference between the actual noise (ε) and the predicted noise (ε_predicted). The model's weights are adjusted to minimize this error.

By training on this noise-prediction task over and over with different images and timesteps, the model learns a general function for removing noise. During generation, this predicted noise is used in a formula to calculate the next, slightly cleaner image in the sequence. While alternative approaches exist, such as predicting the clean image directly, predicting the noise is the most common objective for DDPMs (Denoising Diffusion Probabilistic Models).

An Illustrative Example

Imagine our "image" is just the number 100.

  • Training Step:

    • We pick a random noise value, say ε = 5.
    • For a chosen timestep t, the noising formula gives us a noisy image. Using simplified coefficients for this illustration, the result could be: x_t = 0.9 * 100 + 0.1 * 5 = 90.5.
    • The model receives 90.5 and t as input.
    • Its goal is to output the number 5. If it predicts 4, the training process will nudge its parameters to increase the output next time.
  • Generation Step:

    • The model starts with a noisy input, say x_t = 85.
    • It predicts the noise it thinks is present, say ε_predicted = 6.
    • A denoising formula then uses this prediction to calculate x_{t-1}, which might be 88. This new value is slightly "cleaner" and closer to the data the model was trained on.

How are diffusion models trained vs. used for generation?

A common point of confusion is the difference between the training loop and the generation (or inference) loop. Training is a one-step process that can happen at any random timestep, while generation is a multi-step, sequential process that always goes from T down to 1.

Training Loop (Learning to Denoise) Generation/Inference Loop (Creating an Image)
1. Start with a clean image x₀ from the dataset. 1. Start with a tensor of pure random noise x_T.
2. Pick a random timestep t. 2. Loop for t from T down to 1.
3. Generate a single sample of random noise ε. 3. Feed the current image x_t and the current timestep t to the model.
4. Use x₀, ε, and t to calculate the noisy image x_t in a single step. 4. The model predicts the noise ε_predicted present in x_t.
5. Give x_t and t to the model, which predicts ε_predicted. 5. Use ε_predicted in a formula to calculate the denoised image x_{t-1}.
6. Calculate the loss (error) between the real noise ε and the predicted noise ε_predicted. 6. This denoising step usually involves adding a small amount of new randomness to ensure the output is novel.
7. Update the model's weights to minimize this loss. 7. The output x_{t-1} becomes the input for the next loop iteration.
The goal is to teach the model how to predict the noise for any image at any timestep. The final output x₀ is the generated image. Sampling begins from random noise and moves backward toward a clean data sample (Diffusion Models: A Comprehensive Survey of Methods and Applications).

How does text conditioning guide the process?

A diffusion model can be trained unconditionally to generate random images from its training distribution. To generate images from a text prompt, we need to provide that text as a condition to the model at each step of the reverse process.

Here’s how text-to-image models steer generation:

  1. Text Encoding: The text prompt is converted into a numerical representation, called an embedding, that captures its semantic meaning.

  2. Conditioning the Denoising Model: The text embedding is fed into the denoising neural network at each step. This allows the denoising model to use information from the text prompt when making its noise prediction. The model's task changes from "predict the noise" to "predict the noise that would lead to an image matching this text."

  3. Classifier-Free Guidance (CFG): To improve how well the image matches the prompt, a technique called Classifier-Free Guidance is widely used. At each denoising step, the model actually makes two noise predictions:

    • One prediction is conditional (using the text prompt).
    • The other is unconditional (ignoring the text prompt).

    The final noise prediction is calculated by taking the unconditional prediction and pushing it further in the direction of the conditional prediction. A "guidance scale" parameter controls how strongly the generation adheres to the prompt, often at the cost of some diversity or realism.

Model Variations and Sampling Speed

While the core concept of reversing a noise process is consistent, different implementations exist with various trade-offs.

Alternative Prediction Targets

While predicting the added noise (ε) is common, it is not the only option. Some models are trained to directly predict the original clean image (x₀) from the noisy input x_t. Other variations train the model to predict a related quantity called the "score" or a value used in "v-prediction." These different parameterizations are mathematically related but can affect training stability and the quality of samples generated with fewer steps (Denoising Diffusion Probabilistic Models).

Accelerating Generation

A significant drawback of the original diffusion model formulation was the slow generation speed, which could require hundreds or thousands of denoising steps. To make generation faster, researchers have developed various acceleration techniques. These include DDIM-style samplers, which allow for larger and more efficient steps during generation. Other methods involve using more advanced numerical solvers, "distilling" a slow model into a faster one, or designing schedules that require fewer steps. These methods seek to balance speed with sample quality and diversity.

When are diffusion models not the best choice?

Despite their power, diffusion models have trade-offs and are not always the right tool for every task.

  • When you need real-time generation: The iterative sampling process is inherently slow. While sampling can be accelerated, it still takes many steps, making diffusion models less suitable for applications requiring instant results compared to models that generate in a single pass.
  • When you have limited compute resources: Training and running large diffusion models requires substantial GPU memory and processing power.
  • For tasks requiring perfect data recall: Diffusion models are generative and stochastic. They are designed to create novel samples, not to perfectly reconstruct or retrieve training data. For tasks that function like a database, other architectures are more appropriate.
  • When control over fine details is paramount without iteration: While techniques exist to guide diffusion models, achieving precise, deterministic control over every detail in a single shot is difficult. The multi-step, partially random process makes fine-grained, repeatable edits challenging.

Understanding these limitations helps in choosing the right generative model for a specific problem. For many creative applications where quality is paramount and latency is tolerable, diffusion models offer state-of-the-art performance.

What to learn next

Sources and further reading