Large Language Models (LLMs) seem to generate text one word at a time, but this process relies on a crucial optimization to be fast enough for practical use. This optimization is the Key-Value (KV) cache, a memory-for-speed trade-off at the heart of LLM inference. Understanding the kv cache llm optimization is key to understanding its performance, memory limits, and the techniques used to serve models efficiently.

Key takeaways

  • The KV cache makes text generation practical by storing intermediate calculations—the attention keys (K) and values (V)—from previous tokens.
  • It is a core reason why generating text is a memory-intensive process; the cache's size grows linearly with the number of tokens in the context and the number of requests being processed in a batch.
  • You can calculate the memory cost per token with a simple formula: 2 × layers × KV_heads × head_dim × bytes_per_element.
  • Model architectures like Multi-Query (MQA) and Grouped-Query Attention (GQA) were designed specifically to reduce the size of the KV cache by sharing K and V tensors among attention heads (Source).
  • Modern inference systems use advanced methods like PagedAttention, quantization, and offloading to manage this memory bottleneck.

What Is a KV Cache in an LLM?

The KV cache is a memory storage that holds the Key (K) and Value (V) tensors for all previously processed tokens in a sequence. At each step of text generation, the model reuses this stored information instead of recomputing it. This drastically reduces the amount of computation needed to produce the next token, making autoregressive decoding efficient at the cost of higher GPU memory usage (Source).

To understand the KV cache, you must first understand its role in the self-attention mechanism. For every token, the self-attention layer calculates three vectors: a Query (Q), a Key (K), and a Value (V). The Q vector of the current token is compared against the K vectors of all previous tokens to calculate attention scores. These scores determine how much focus to place on each of those previous tokens. The scores are then used to create a weighted sum of the V vectors, producing the final output for the current token. The cache tensors for the keys and values are often conceptualized with a shape of [batch_size, num_heads, sequence_length, head_dim] for each layer (Source).

Without a KV cache, generating the 100th token would require recalculating the K and V vectors for tokens 1 through 99. Then, to generate the 101st token, it would have to recalculate the K and V vectors for tokens 1 through 100 all over again. This process would be incredibly slow and computationally expensive.

The KV cache solves this by storing the K and V vectors for each token at every layer as they are computed. When generating the next token, the model only computes the Q, K, and V vectors for the single new token. It then appends the new K and V vectors to the cache and uses the new Q vector to attend to the entire history of keys stored in the cache (Source). This turns a computationally intensive process into a memory-intensive one.

How Does the KV Cache Work, Step by Step?

The KV cache is used in a two-phase process: prefill (processing the prompt) and decoding (generating new tokens). This separation is fundamental to the process of how LLM inference works.

Phase 1: Prefill (Prompt Processing)

In this phase, the model processes the entire input prompt at once.

  1. The LLM receives the prompt, for example, "The capital of France".
  2. All tokens ("The", "capital", "of", "France") are fed into the model in parallel.
  3. At each self-attention layer in the Transformer architecture, the model computes Q, K, and V vectors for every one of these tokens.
  4. The K and V vectors for all prompt tokens are calculated and stored in the newly created KV cache. The cache is now "filled" with the context of the prompt.

After this step, the cache for a single layer might conceptually look like this:

  • Key Cache: [K_The, K_capital, K_of, K_France]
  • Value Cache: [V_The, V_capital, V_of, V_France]

This happens for every attention layer in the model.

Phase 2: Decode (Token Generation)

In this phase, the model generates text one token at a time, reusing and extending the cache at each step.

  1. Generate Token 1 (e.g., "is"):

    • The model takes the last token from the input ("France") to predict the next token.
    • It computes only the Q, K, and V vectors for this new prediction step. Let's call them Q_new, K_new, and V_new.
    • The new K_new and V_new are appended to the cache. The cache now holds keys for "The", "capital", "of", "France", and the new token being generated.
    • The new Q_new attends to the entire history of keys in the updated cache.
    • Based on this attention output, the model predicts the token "is".
  2. Generate Token 2 (e.g., "Paris"):

    • The model now takes the previously generated token ("is") as input.
    • It computes a new set of Q, K, and V vectors just for this step.
    • It appends the new K and V vectors to the cache, which now contains the context for "The capital of France is".
    • The new Q vector attends to this full, expanded cache.
    • The model predicts the next token, "Paris".

This cycle repeats. At every step, the model performs a small amount of computation for the new token and re-reads the entire, ever-growing KV cache. This is what makes decoding a memory-bandwidth-bound operation.

How Much GPU Memory Does a KV Cache Use per Token?

You can calculate the GPU memory required for the KV cache with a straightforward formula. The cache size depends on the model's architecture, the number of tokens being stored, the batch size (how many sequences are processed in parallel), and the numerical precision used. The size grows linearly with sequence length and batch size.

The formula for the memory required to store one token's KV pairs across all layers is:

Cache Memory per Token = 2 × num_layers × num_KV_heads × head_dimension × bytes_per_element

Let's break down each component:

  • 2: This accounts for storing both a Key and a Value tensor.
  • num_layers: The number of transformer layers in the model. Each layer has its own cache (Source).
  • num_KV_heads: The number of Key/Value heads in the attention mechanism. Important: This is not always the same as the number of Query heads (more on this with MQA/GQA later).
  • head_dimension: The size of each head's K and V vector.
  • bytes_per_element: The memory size of each number in the tensor. This is typically 2 for 16-bit floating point (FP16) or 1 for 8-bit integer quantization (INT8) (Source).

Worked Example: A Hypothetical Model

Let's calculate the KV cache memory for a hypothetical model that uses Grouped-Query Attention (GQA).

Illustrative Parameters:

  • num_layers: 32
  • num_KV_heads: 8
  • head_dimension: 128
  • bytes_per_element: 2 (for FP16 precision)

Calculation per Token:

  • Memory per Token = 2 × 32 × 8 × 128 × 2 bytes
  • Memory per Token = 131,072 bytes, or 128 KB

Now, let's calculate the total KV cache size for a single sequence with an 8,192-token context length:

  • Total Cache Size = 8,192 tokens × 128 KB/token
  • Total Cache Size = 1,048,576 KB, or 1 GB

If you are serving a batch of 16 such requests simultaneously, the memory requirement explodes:

  • Total Batch Cache Size = 1 GB/request × 16 requests
  • Total Batch Cache Size = 16 GB

This 16 GB is in addition to the memory needed for the model weights, activations, and other overhead like CUDA kernels, allocator fragmentation, and workspace memory, showing how the KV cache quickly becomes a dominant factor in GPU memory consumption.

Why Does the KV Cache Make Long-Context Decoding Memory-Bound?

The KV cache fundamentally changes the performance bottleneck during inference. While processing the initial prompt (prefill) is often limited by the GPU's raw computation speed (FLOPs), generating subsequent tokens (decode) is limited by its memory bandwidth.

During prefill, the model performs large matrix multiplications across all input tokens in parallel, which is computationally intensive. The GPU is busy doing math.

During each decode step, the model only computes Q, K, and V for a single new token. This is a very small amount of computation. The main operation is reading the entire KV cache from GPU memory to calculate the attention scores for that one new token (Source). As the sequence gets longer, the cache grows larger, and the time it takes to read it from memory becomes the limiting factor. The GPU is not waiting on math; it is waiting on data.

This is why token generation speed (tokens per second) often decreases as the context length grows. The memory bandwidth bottleneck becomes more severe because a larger and larger cache must be read at every single step.

How Do MQA and GQA Reduce KV Cache Memory?

Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-Query Attention (MQA) are different architectural designs that directly impact the size of the KV cache. The key difference is how many Key and Value heads are shared among the Query heads.

  • Multi-Head Attention (MHA): This is the original design. Every Query (Q) head has its own unique Key (K) and Value (V) head. This provides maximum representational power but also results in the largest possible KV cache.
  • Grouped-Query Attention (GQA): This is a compromise. It groups several Q heads together and has them share a single K and V head. This reduces the number of K/V heads, shrinking the cache size and memory bandwidth requirements while retaining much of the model quality (Source).
  • Multi-Query Attention (MQA): This is the most extreme optimization. All Q heads share a single K and V head. This results in the smallest possible KV cache but can sometimes lead to a drop in model quality compared to MHA or GQA (Source).

The table below compares these approaches for a hypothetical model with 32 query heads.

Attention Type Query (Q) Heads Key/Value (K/V) Heads K/V to Q Ratio Relative Cache Size
MHA 32 32 1:1 100%
GQA (8 groups) 32 8 1:4 25%
MQA 32 1 1:32 ~3%

As you can see, choosing GQA or MQA during model design is a critical decision that trades a small amount of model quality for a massive reduction in KV cache memory, enabling longer context and higher throughput.

What Are the Common KV Cache Optimization Techniques?

Beyond architectural choices like GQA, several techniques are used in modern serving systems to manage the KV cache memory bottleneck.

  • PagedAttention: This technique, inspired by virtual memory in operating systems, allocates the KV cache in non-contiguous, fixed-size blocks called "pages." Instead of reserving a single large chunk of memory for each request, memory is allocated on demand as tokens are generated. This virtually eliminates memory fragmentation and allows for much higher GPU utilization (Source). Alternatives like vAttention also seek to improve memory management.
  • Quantization: This reduces the numerical precision of the tensors stored in the cache. For example, converting 16-bit floating-point numbers (FP16) to 8-bit integers (INT8) cuts the cache size in half. While this can risk a small loss in accuracy, it is often a worthwhile trade-off for the memory savings (Source).
  • Static vs. Dynamic Cache Allocation: Systems can handle cache memory in two ways. A static cache pre-allocates a large, contiguous block of memory for each request, up to the maximum context length. This can simplify memory management and is compatible with some compilation-based optimizations. A dynamic cache allocates memory as needed while tokens are being generated, often resulting in less wasted memory, especially for requests that don't reach the maximum context length (Source). PagedAttention is an advanced form of dynamic allocation.
  • Offloading: For extremely long sequences that do not fit in GPU memory, parts of the KV cache can be "offloaded" to CPU RAM or even NVMe storage. This allows for massive context lengths but introduces significant latency, as the data must be moved back to the GPU when needed.
  • Sliding Window Attention: Instead of allowing the cache to grow indefinitely, this method only keeps the K and V pairs for the most recent N tokens (e.g., the last 4096 tokens). This puts a fixed upper bound on the cache size but prevents the model from attending to tokens outside this window.

What Are the Common Mistakes and Limits to Know About?

When working with the KV cache, beginners often run into a few common pitfalls.

  • Mistake: Using the number of query heads instead of K/V heads in memory calculations. For models with GQA or MQA, these numbers are different, and using the wrong one will lead to a significant overestimation of memory usage.
  • Mistake: Forgetting the factor of 2 in the memory formula. The cache stores both Keys and Values.
  • Mistake: Confusing a model's maximum context window with the memory available to use it. A model might support a 32k context, but you may not have enough GPU memory to handle that length, especially with a large batch size.
  • Mistake: Confusing a static cache's pre-allocated size with actual token usage. A static cache reserves memory for the maximum possible sequence length, even if the current sequence is short. This reserved but unused memory is a common source of confusion when analyzing memory usage.
  • Mistake: Assuming cache optimizations like quantization are lossless. They are a trade-off between memory and potential precision loss.
  • Limit: The KV cache is request-specific. You cannot typically share the cache between two different users asking two different questions, as the cached history is unique to each conversation (Source).
  • Limit: The cache's memory cost is in addition to model weights and activations. When diagnosing out-of-memory errors, you must account for all sources of memory consumption, including padding, memory fragmentation, and temporary buffers used by attention kernels (Source).

What to learn next

Sources and further reading