Large language models (LLMs) generate text one token at a time in a process called autoregressive decoding. This sequential nature means generating a long response can be slow, as each token requires a full, time-consuming pass through the massive model. Speculative decoding is a technique that speeds up this process without sacrificing output quality.
It works by using a small, fast "draft" model to predict a short sequence of future tokens. Then, the large, powerful "target" model checks all these proposed tokens at once in a single pass. This "draft-then-verify" approach allows the system to generate multiple tokens for the cost of one large model pass, significantly reducing the overall time it takes to generate a response.
Key takeaways
- Speculative decoding accelerates LLM inference by using a small "draft" model to generate candidate tokens and a large "target" model to verify them in parallel.
- It is not an approximation. The method produces an output distribution that is mathematically identical to what the target model would have produced on its own (Source name).
- The speedup comes from generating several accepted tokens for the cost of a single target model pass. The final speedup depends on factors including memory bandwidth, batch size, and the models used.
- Performance gains are not guaranteed. They depend on the draft model's quality (acceptance rate), the overhead of the drafting process, and the hardware used.
- Modern variations include Medusa, which uses auxiliary prediction heads instead of a separate draft model, and dynamic lookahead methods, which adapt the number of drafted tokens (Source name).
What Is Speculative Decoding?
Speculative decoding is an optimization method that reduces the latency of text generation in large language models. It uses a small, fast draft model to propose a sequence of several future tokens, which the main large model then verifies in a single parallel batch. This allows the system to produce multiple tokens per single pass of the large model, speeding up inference while maintaining the exact same output quality.
Standard autoregressive decoding is a slow, sequential process. As explained in our guide on how LLM inference works, generating each new token requires a full forward pass of the entire model. Speculative decoding breaks this one-pass-per-token bottleneck.
The core idea is to use a cheaper model to make an educated guess about the next few tokens. The expensive model then evaluates this guess efficiently. Because modern GPUs perform parallel processing efficiently, verifying multiple tokens at once is highly efficient compared to generating each token sequentially. If the draft model's guesses are good, you get several tokens for the price of one.
How Does Speculative Decoding Work, Step by Step?
Speculative decoding works by following a five-step "draft, verify, accept/reject, correct, repeat" cycle for each block of tokens. A small draft model makes a guess, and a large target model checks it. This process ensures the output is identical to what the target model would have produced alone, just generated faster. The number of tokens guessed in each cycle is often called the lookahead or gamma (γ).
Here is a breakdown of the algorithm:
- Draft. Start with the existing sequence of tokens. A small, fast draft model autoregressively generates a short continuation of γ new tokens. For example, if γ is 4, the draft model generates 4 tokens one by one.
- Verify. The large target model receives the original sequence plus all γ drafted tokens as a single input. It performs one forward pass to calculate the correct probability distributions for what the next token should be at each of the γ+1 positions, all in parallel (Source name).
- Accept/Reject. The algorithm iterates through the drafted tokens one by one, comparing the draft's choice with the target's verified probabilities. A rejection-sampling rule based on the target and draft model probabilities determines whether to accept or reject each token. The process stops at the first rejection.
- Correct. If a drafted token is rejected, it is discarded. A new token is then sampled from the target model's corrected probability distribution for that position (Source name). This step guarantees the final output remains true to the target model.
- Repeat. The sequence of all accepted tokens, plus the newly sampled token (if one was corrected), is appended to the main output. This new, longer sequence becomes the starting point for the next draft-and-verify cycle.
This process involves managing state for both models, including their respective attention caches, often called the KV cache.
A Worked Example: Tracing a Single Generation Step
To make this concrete, let's trace a single step using greedy decoding, where we always pick the token with the highest probability.
Imagine the LLM has already generated the phrase The old clock chimed at. We want to generate the next few tokens. Let's set our lookahead (γ) to 3 tokens.
Step 1: Draft
The small draft model starts with ...chimed at and autoregressively generates a 3-token guess.
- It predicts the next token is
midnight. - Then, given
...at midnight, it predictssharp. - Finally, given
...at midnight sharp, it predicts..
So, the draft proposal is the sequence: midnight, sharp, ..
Step 2: Verify
The large target model now gets the full sequence The old clock chimed at midnight sharp . as a single input. It runs a single forward pass and computes its own probabilities for the three token positions after at.
Let's say the target model's top choices (highest probability tokens) are:
- For the position after
at:midnight - For the position after
midnight:sharp - For the position after
sharp:the
Step 3 & 4: Accept/Reject and Correct
Now, we compare the draft's proposals to the target's verified choices, one by one.
Position 1:
- Draft proposed:
midnight - Target verified:
midnight - They match. ACCEPT.
- Draft proposed:
Position 2:
- Draft proposed:
sharp - Target verified:
sharp - They match. ACCEPT.
- Draft proposed:
Position 3:
- Draft proposed:
. - Target verified:
the - They do not match. REJECT.
- Draft proposed:
The loop stops at the first rejection. We discard the draft's proposal (.) and instead use the target model's verified token (the).
Step 5: Repeat
The tokens generated in this step are the ones we accepted (midnight, sharp) plus the corrected one (the).
The final output is now: The old clock chimed at midnight sharp the.
In this single cycle, we successfully generated three tokens but only ran the expensive target model once. Without speculative decoding, this would have required three separate, slow passes. This is where the speedup comes from.
Why Is the Output Identical to Normal Decoding?
Speculative decoding produces mathematically identical output to the target model because it uses a form of rejection sampling to correct for any differences between the draft and target models. This ensures that even though a draft model proposes tokens, the final selection strictly follows the target model's probability distribution. It is not an approximation (Source name).
For greedy decoding, the logic is simple. The rule is to accept a drafted token only if it is the same one the target model would have chosen. The first time they disagree, the draft token is thrown away and replaced with the target model's choice (Source name). This guarantees the sequence is exactly what the target model would have generated greedily.
For stochastic sampling (e.g., using temperature), the process is more sophisticated but provides the same guarantee. The acceptance and correction process relies on a rejection-sampling rule based on the probabilities from both models. If a proposed token is rejected, a replacement token is sampled from a corrected 'residual' distribution. This residual distribution is formed by subtracting the draft model's probabilities from the target model's. This procedure ensures the final token distribution matches the target model's exactly (Source name).
Variations of Speculative Decoding
Speculative decoding is a general concept, and researchers have developed several variations. The original method uses two separate models, but newer approaches like Medusa integrate the drafting process into the target model itself. Dynamic lookahead methods focus on optimizing how many tokens to speculate.
Here is a comparison of common approaches:
| Method | Proposal Mechanism | Key Feature |
|---|---|---|
| Classic Speculative Decoding | A separate, smaller draft LLM. | Requires deploying and maintaining two separate models (Source name). |
| Medusa | Multiple prediction "heads" attached to the target model. | Avoids a separate draft model. Proposes and verifies candidates in a tree structure for higher efficiency (Source name). |
| Dynamic Lookahead Methods | Uses a base speculative method, such as a draft model. | Focuses on adapting the number of speculative tokens (lookahead) per step, instead of using a single fixed value (Source name). |
While classic speculative decoding laid the groundwork, methods like Medusa are gaining popularity because they avoid the complexity of managing a second model. Dynamic lookahead methods further refine the process by adapting the speculation length on the fly, as the optimal number of tokens to guess can change during generation (Source name).
What Are the Common Mistakes and Limits to Know About?
Speculative decoding is a powerful technique, but it is not a universal solution. The performance gains depend heavily on the implementation, workload, and models involved. Simply turning it on does not guarantee a speedup.
Here are some common pitfalls and limitations:
- Assuming more speculation is always better. Guessing too many tokens (a long γ) can be counterproductive. The probability of the target model accepting a long, unbroken sequence of draft tokens is low. This can lead to high rejection rates, where the overhead of drafting outweighs the benefit (Source name).
- Ignoring the draft model's cost. The draft model must be significantly faster than the target model. If it is too large or slow, or if the overhead of running it is too high, it can erase any potential gains.
- Low acceptance rates. The method works best when the draft model's predictions align with the target's. For tasks requiring complex reasoning or code generation, the draft model may struggle, leading to a low acceptance rate and minimal speedup.
- Implementation overhead. Managing two models, their KV caches, and the complex verification logic adds significant complexity to an inference server. This can introduce bugs or performance bottlenecks, especially in high-throughput environments using techniques like continuous batching.
- Comparing speedups without context. A claim like "2x speedup" is meaningless without details. The actual performance depends on the hardware (GPU), batch size, models used, and the specific task. An observed speedup in one scenario may not apply to another. For example, some early results showed a 2x–2.5x improvement on specific models and hardware (Source name).
What to learn next
- Semantic Caching LLM: A Production Guide - Learn how a semantic caching LLM system reduces cost and latency. This guide covers safe implementation, choosing thresholds, and avoiding c
- On Device LLM: The Complete How-It-Works Guide - Learn how an on device LLM runs locally on your phone or laptop. This guide explains memory needs, GGUF, llama.cpp, and the trade-offs of ed
- LLM Gateway: A Guide to Smart Model Routing - A complete guide to using an LLM gateway. Learn how model routing works, how to route by cost and difficulty, and the common pitfalls to avo
- Tensor Parallelism vs. Pipeline Parallelism: How to Choose - A deep dive into tensor parallelism vs pipeline parallelism. Learn how each shards LLMs, their trade-offs in communication and latency, and
- LLM Cost Per Token: A Practical Calculation Guide - Learn to calculate the true LLM cost per token for APIs and self-hosted models. This guide provides step-by-step formulas for inference cost
- Continuous Batching LLM: How It Works & Boosts Throughput: Explore another key technique for improving LLM inference throughput by dynamically managing requests in a batch.
- The KV Cache LLM Explained: A Complete Technical Guide: Understand the memory optimization that makes generating long sequences with Transformer models practical.
- How LLM Inference Works: From Prompt to Token Stream: Get a foundational understanding of the entire text generation process, from input prompt to the final output.
Sources and further reading
- Fast Inference from Transformers via Speculative Decoding
- Accelerating Large Language Model Decoding with Speculative Sampling
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
- Scaling Speculative Decoding with Lookahead Reasoning
