Large language models (LLMs) generate text one token at a time in a process called autoregressive decoding. This sequential nature means generating a long response can be slow, as each token requires a full, time-consuming pass through the massive model. Speculative decoding is a technique that speeds up this process without sacrificing output quality.

It works by using a small, fast "draft" model to predict a short sequence of future tokens. Then, the large, powerful "target" model checks all these proposed tokens at once in a single pass. This "draft-then-verify" approach allows the system to generate multiple tokens for the cost of one large model pass, significantly reducing the overall time it takes to generate a response.

Key takeaways

  • Speculative decoding accelerates LLM inference by using a small "draft" model to generate candidate tokens and a large "target" model to verify them in parallel.
  • It is not an approximation. The method produces an output distribution that is mathematically identical to what the target model would have produced on its own (Source name).
  • The speedup comes from generating several accepted tokens for the cost of a single target model pass. The final speedup depends on factors including memory bandwidth, batch size, and the models used.
  • Performance gains are not guaranteed. They depend on the draft model's quality (acceptance rate), the overhead of the drafting process, and the hardware used.
  • Modern variations include Medusa, which uses auxiliary prediction heads instead of a separate draft model, and dynamic lookahead methods, which adapt the number of drafted tokens (Source name).

What Is Speculative Decoding?

Speculative decoding is an optimization method that reduces the latency of text generation in large language models. It uses a small, fast draft model to propose a sequence of several future tokens, which the main large model then verifies in a single parallel batch. This allows the system to produce multiple tokens per single pass of the large model, speeding up inference while maintaining the exact same output quality.

Standard autoregressive decoding is a slow, sequential process. As explained in our guide on how LLM inference works, generating each new token requires a full forward pass of the entire model. Speculative decoding breaks this one-pass-per-token bottleneck.

The core idea is to use a cheaper model to make an educated guess about the next few tokens. The expensive model then evaluates this guess efficiently. Because modern GPUs perform parallel processing efficiently, verifying multiple tokens at once is highly efficient compared to generating each token sequentially. If the draft model's guesses are good, you get several tokens for the price of one.

How Does Speculative Decoding Work, Step by Step?

Speculative decoding works by following a five-step "draft, verify, accept/reject, correct, repeat" cycle for each block of tokens. A small draft model makes a guess, and a large target model checks it. This process ensures the output is identical to what the target model would have produced alone, just generated faster. The number of tokens guessed in each cycle is often called the lookahead or gamma (γ).

Here is a breakdown of the algorithm:

  1. Draft. Start with the existing sequence of tokens. A small, fast draft model autoregressively generates a short continuation of γ new tokens. For example, if γ is 4, the draft model generates 4 tokens one by one.
  2. Verify. The large target model receives the original sequence plus all γ drafted tokens as a single input. It performs one forward pass to calculate the correct probability distributions for what the next token should be at each of the γ+1 positions, all in parallel (Source name).
  3. Accept/Reject. The algorithm iterates through the drafted tokens one by one, comparing the draft's choice with the target's verified probabilities. A rejection-sampling rule based on the target and draft model probabilities determines whether to accept or reject each token. The process stops at the first rejection.
  4. Correct. If a drafted token is rejected, it is discarded. A new token is then sampled from the target model's corrected probability distribution for that position (Source name). This step guarantees the final output remains true to the target model.
  5. Repeat. The sequence of all accepted tokens, plus the newly sampled token (if one was corrected), is appended to the main output. This new, longer sequence becomes the starting point for the next draft-and-verify cycle.

This process involves managing state for both models, including their respective attention caches, often called the KV cache.

A Worked Example: Tracing a Single Generation Step

To make this concrete, let's trace a single step using greedy decoding, where we always pick the token with the highest probability.

Imagine the LLM has already generated the phrase The old clock chimed at. We want to generate the next few tokens. Let's set our lookahead (γ) to 3 tokens.

Step 1: Draft

The small draft model starts with ...chimed at and autoregressively generates a 3-token guess.

  • It predicts the next token is midnight.
  • Then, given ...at midnight, it predicts sharp.
  • Finally, given ...at midnight sharp, it predicts ..

So, the draft proposal is the sequence: midnight, sharp, ..

Step 2: Verify

The large target model now gets the full sequence The old clock chimed at midnight sharp . as a single input. It runs a single forward pass and computes its own probabilities for the three token positions after at.

Let's say the target model's top choices (highest probability tokens) are:

  • For the position after at: midnight
  • For the position after midnight: sharp
  • For the position after sharp: the

Step 3 & 4: Accept/Reject and Correct

Now, we compare the draft's proposals to the target's verified choices, one by one.

  • Position 1:

    • Draft proposed: midnight
    • Target verified: midnight
    • They match. ACCEPT.
  • Position 2:

    • Draft proposed: sharp
    • Target verified: sharp
    • They match. ACCEPT.
  • Position 3:

    • Draft proposed: .
    • Target verified: the
    • They do not match. REJECT.

The loop stops at the first rejection. We discard the draft's proposal (.) and instead use the target model's verified token (the).

Step 5: Repeat

The tokens generated in this step are the ones we accepted (midnight, sharp) plus the corrected one (the).

The final output is now: The old clock chimed at midnight sharp the.

In this single cycle, we successfully generated three tokens but only ran the expensive target model once. Without speculative decoding, this would have required three separate, slow passes. This is where the speedup comes from.

Why Is the Output Identical to Normal Decoding?

Speculative decoding produces mathematically identical output to the target model because it uses a form of rejection sampling to correct for any differences between the draft and target models. This ensures that even though a draft model proposes tokens, the final selection strictly follows the target model's probability distribution. It is not an approximation (Source name).

For greedy decoding, the logic is simple. The rule is to accept a drafted token only if it is the same one the target model would have chosen. The first time they disagree, the draft token is thrown away and replaced with the target model's choice (Source name). This guarantees the sequence is exactly what the target model would have generated greedily.

For stochastic sampling (e.g., using temperature), the process is more sophisticated but provides the same guarantee. The acceptance and correction process relies on a rejection-sampling rule based on the probabilities from both models. If a proposed token is rejected, a replacement token is sampled from a corrected 'residual' distribution. This residual distribution is formed by subtracting the draft model's probabilities from the target model's. This procedure ensures the final token distribution matches the target model's exactly (Source name).

Variations of Speculative Decoding

Speculative decoding is a general concept, and researchers have developed several variations. The original method uses two separate models, but newer approaches like Medusa integrate the drafting process into the target model itself. Dynamic lookahead methods focus on optimizing how many tokens to speculate.

Here is a comparison of common approaches:

Method Proposal Mechanism Key Feature
Classic Speculative Decoding A separate, smaller draft LLM. Requires deploying and maintaining two separate models (Source name).
Medusa Multiple prediction "heads" attached to the target model. Avoids a separate draft model. Proposes and verifies candidates in a tree structure for higher efficiency (Source name).
Dynamic Lookahead Methods Uses a base speculative method, such as a draft model. Focuses on adapting the number of speculative tokens (lookahead) per step, instead of using a single fixed value (Source name).

While classic speculative decoding laid the groundwork, methods like Medusa are gaining popularity because they avoid the complexity of managing a second model. Dynamic lookahead methods further refine the process by adapting the speculation length on the fly, as the optimal number of tokens to guess can change during generation (Source name).

What Are the Common Mistakes and Limits to Know About?

Speculative decoding is a powerful technique, but it is not a universal solution. The performance gains depend heavily on the implementation, workload, and models involved. Simply turning it on does not guarantee a speedup.

Here are some common pitfalls and limitations:

  • Assuming more speculation is always better. Guessing too many tokens (a long γ) can be counterproductive. The probability of the target model accepting a long, unbroken sequence of draft tokens is low. This can lead to high rejection rates, where the overhead of drafting outweighs the benefit (Source name).
  • Ignoring the draft model's cost. The draft model must be significantly faster than the target model. If it is too large or slow, or if the overhead of running it is too high, it can erase any potential gains.
  • Low acceptance rates. The method works best when the draft model's predictions align with the target's. For tasks requiring complex reasoning or code generation, the draft model may struggle, leading to a low acceptance rate and minimal speedup.
  • Implementation overhead. Managing two models, their KV caches, and the complex verification logic adds significant complexity to an inference server. This can introduce bugs or performance bottlenecks, especially in high-throughput environments using techniques like continuous batching.
  • Comparing speedups without context. A claim like "2x speedup" is meaningless without details. The actual performance depends on the hardware (GPU), batch size, models used, and the specific task. An observed speedup in one scenario may not apply to another. For example, some early results showed a 2x–2.5x improvement on specific models and hardware (Source name).

What to learn next

Sources and further reading