When you type a prompt into a large language model (LLM) and see the response appear word by word, you are watching LLM inference in action. This process of generating new text from an input is not a single, uniform task. It is a tale of two distinct phases that determine your application's speed, cost, and feel.

Understanding how LLM inference works reveals why there is a pause before the first word appears and how the rest of the answer streams in. It explains why a long prompt and a long answer create different kinds of performance challenges.

Key takeaways

  • LLM inference is a two-phase process: a parallel prefill phase for the prompt, followed by a sequential decode phase for generating the response one token at a time.
  • Prefill is typically compute-bound and determines the "time to first token" (TTFT), while the decode phase is memory-bound and determines the generation speed of subsequent tokens (Source).
  • The KV cache is an essential component that stores intermediate calculations. It makes generation efficient by avoiding re-computation, but it also consumes a large and growing amount of GPU memory (Source).
  • Autoregressive generation is the core loop of the decode phase, where the model feeds its own output back as input to produce the next token in the sequence.
  • Token streaming improves perceived performance by sending each token to the user as it's generated, creating the familiar word-by-word effect.

What Is LLM Inference? A Two-Phase Process

LLM inference is the process of using a trained model to generate new output from an input prompt. It has two main phases: prefill and decode (Source). This is different from training, which is the much larger, one-time process of creating the model's parameters using a massive dataset.

Inference starts when you provide a prompt. The model doesn't just process this text in one go. Instead, it splits the work.

  1. Prefill Phase: The model processes all the tokens in your prompt simultaneously in a single, parallel step. This initial computation is intensive and is the main contributor to the delay you experience before seeing the first word of the response.
  2. Decode Phase: After processing the prompt, the model enters a loop to generate the response. In each step of this loop, it generates just one token, appends it to the sequence, and uses that new sequence to generate the next token. This sequential, one-at-a-time process is known as autoregressive generation.

This separation is the key to understanding LLM performance. The length of your prompt affects the prefill phase, while the length of the generated answer affects the decode phase (Source).

How LLM Inference Works, Step by Step

LLM inference turns your text prompt into a generated response through a structured, multi-step process. For models based on the Transformer Architecture Explained: From Tokens to Logits, this involves a parallel prompt processing step followed by a sequential generation loop. Here is how it works from start to finish.

  1. Tokenization: First, your text prompt is broken down into smaller pieces called tokens. A token is an integer ID that represents a word, a part of a word, or punctuation. For example, the text "Hello, world!" might become four tokens: [15496, 11, 995, 0].

  2. Prefill (Prompt Processing): The model performs a single, large computation called a forward pass on all the prompt tokens at once. During this step, the model's attention layers calculate intermediate values called keys (K) and values (V) for every token in the prompt. These key-value pairs are stored in a dedicated block of GPU memory called the KV cache (Source). This step ends by predicting the very first token of the output.

  3. Next-Token Selection: The prefill step produces a list of scores, called logits, for every possible token in the model's vocabulary. A sampling method then converts these logits into a single chosen token. The simplest method is greedy sampling, which just picks the token with the highest score. Other methods like top-p sampling introduce randomness to produce more creative responses.

  4. Decode (Autoregressive Generation): The single token chosen in the previous step is now used as input for the next forward pass. The model calculates this new token's attention query (Q) and reuses the keys and values for all previous tokens from the KV cache. This avoids re-computing the entire sequence (Source). The model generates a new key and value for the current token, adds them to the cache, and produces logits for the next token.

  5. Repeat or Stop: The decode loop (Step 4) repeats, generating one token per iteration (Source). If token streaming is enabled, each newly generated token is sent to the user immediately. The loop continues until the model generates a special stop token, reaches a configured length limit, or is stopped by another condition.

  6. Detokenization: Finally, the sequence of generated token IDs is converted back into human-readable text and presented as the final response.

Why Are Prefill and Decode Different Workloads?

Prefill and decode are different workloads because they process different numbers of tokens at once, leading to distinct performance bottlenecks. This difference directly maps to key performance metrics: prefill latency determines the Time to First Token (TTFT), while decode speed governs the Time Between Tokens (TBT) (Source). Prefill processes many prompt tokens in parallel and is typically limited by the GPU's computational power. Decode generates one token at a time and is often limited by the speed of moving data from GPU memory (Source).

A long prompt creates a heavy prefill workload, causing a noticeable delay before the first token appears. A long answer requires many decode steps, which can feel slow if the memory bandwidth is a bottleneck.

This table compares the two phases:

Characteristic Prefill (Prompt Processing) Decode (Token Generation)
Input Size Many tokens (the entire prompt) One token at a time
Computation Type Large matrix-matrix multiplications Smaller matrix-vector operations
GPU Bottleneck Compute-bound (limited by raw processing power) Memory-bound (limited by GPU memory speed)
Performance Metric Time to First Token (TTFT) Time Between Tokens (TBT) or Tokens per Second
KV Cache Role Creates the initial cache for all prompt tokens Reads the entire cache and appends one new entry per step

This distinction is so fundamental that advanced serving systems may even run prefill and decode on separate, specialized hardware to optimize resource use (Source).

Tracing a Prompt: A Worked Example

To make these concepts concrete, let's trace a simple prompt through the inference process. The token IDs used here are for illustration only.

Prompt: "Write a haiku"

  1. Tokenization: The text is converted into a sequence of integer IDs.

    • "Write a haiku" -> [29948, 17, 49764]
  2. Prefill: The model processes all three tokens (29948, 17, 49764) in a single parallel forward pass.

    • It calculates key and value vectors for position 1 ("Write"), position 2 ("a"), and position 3 ("haiku").
    • These three key-value pairs are stored in the KV cache.
    • The model outputs logits that predict the token for position 4.
  3. Next-Token Selection: The sampling process selects the most likely next token from the logits.

    • Let's say the chosen token is "An" with ID 501.
  4. First Decode Step: The model begins the autoregressive loop, using only the new token 501 as its direct input.

    • It calculates a new query vector for the token "An" at position 4.
    • It attends to the keys and values for positions 1, 2, and 3 by reading them from the KV cache.
    • It calculates a new key-value pair for "An" and appends it to the cache. The cache now holds four entries.
    • The model outputs new logits predicting the token for position 5. Let's say it selects "ancient" (ID 12128).
  5. Subsequent Decode Steps: This process repeats. For the next step, the model inputs token 12128, reads the four previous entries from the KV cache, adds a fifth entry, and predicts the sixth token. This continues until a stop token is generated.

  6. Detokenization: The sequence of generated token IDs—for example, [501, 12128, 10730, 11, ...]—is converted back into text: "An ancient pond, a frog jumps in..."

What Is the KV Cache and Why Does It Matter?

The KV cache is a performance optimization that stores intermediate calculations from the attention mechanism, but it is also a primary consumer of GPU memory. It stores the "key" and "value" vectors for every previously processed token in a sequence. Without it, generating each new token would require re-calculating attention over the entire history, making inference prohibitively slow.

How it helps: The cache makes the decode phase feasible. By storing the keys and values from the prompt and all previously generated tokens, the model only needs to compute the attention query for the single new token and compare it against the cached history (Source). This significantly reduces repeated computation for each new token by reusing past attention calculations, making decode feasible for longer sequences.

Why it's a challenge: The KV cache's size grows with every token generated. Its memory footprint is determined by the batch size (how many users are being served at once), the sequence length (prompt + generated text), the model size, and its architecture. For models with long context windows or in high-throughput systems, the KV cache can consume many gigabytes of GPU memory, often becoming the main factor limiting how many requests can be handled concurrently (Source).

Managing this memory is a central problem in LLM serving. Techniques like quantization or paged attention are used to reduce the cache's memory footprint and improve system throughput.

Common Pitfalls for Beginners

When first learning about LLM inference, it's easy to fall into a few common misconceptions. Being aware of these can help clarify how the system truly works.

  • Treating tokens as words: A token is not always a full word. It can be a word, a part of a word (like "ing"), a punctuation mark, or even a single space. The specific tokenization rules depend on the model used.
  • Confusing token count with word count: Because of how tokenization works, the number of tokens in a sequence is not the same as the number of words. This is important for understanding model context limits and pricing.
  • Assuming a uniform workload: A common mistake is to think inference is one single, repeated process. In reality, the compute-bound prefill and memory-bound decode phases have different hardware bottlenecks and performance characteristics.
  • Ignoring the KV cache's memory cost: The KV cache is crucial for performance, but its memory footprint grows with every token. Ignoring this can lead to out-of-memory errors and limit how many concurrent users a system can serve, especially with long context lengths.

Conclusion: Two Phases, One Process

LLM inference is a sophisticated process defined by its two distinct phases: a parallel, compute-heavy prefill for the prompt and a sequential, memory-intensive decode for generation. This division explains why the time to first token is different from the time between subsequent tokens and why prompt length and output length create different system loads.

At the heart of this process is the KV cache, an essential optimization that makes autoregressive generation efficient but introduces a significant memory management challenge. By understanding these mechanics—from tokenization to the final detokenized output—you can better diagnose performance issues, optimize costs, and build more responsive and scalable AI applications.

What to learn next

Sources and further reading