When evaluating hardware for AI, many people focus on a GPU's computational power, measured in TFLOP/s (trillion floating-point operations per second). Yet for many modern AI workloads, especially large language model (LLM) inference, another specification is often the true performance bottleneck: GPU memory bandwidth. This is the speed at which a GPU can move data, and it fundamentally limits how fast certain tasks can run.
Understanding this limit is key to building efficient AI systems. This guide explains what GPU memory bandwidth is, how it relates to compute power, and why it is so critical for tasks like generating text with an LLM.
Key takeaways
- Bandwidth vs. Compute: GPU performance is limited by one of two factors: memory bandwidth (the speed of data movement) or compute throughput (the speed of calculations). The bottleneck depends on the workload's "arithmetic intensity"—its ratio of calculations to data moved.
- LLM Decode is Memory-Bound: Autoregressive LLM decoding, which generates text one token at a time, repeatedly reads large model weights from memory. This process has low arithmetic intensity, making memory bandwidth the primary performance limiter, not the GPU's peak TFLOP/s (Source).
- The Roofline Model: This visual model helps identify whether a workload is memory-bound or compute-bound. Performance is capped by a "roof" set by memory bandwidth at low intensity and by compute power at high intensity (Source).
- Peak vs. Achieved Bandwidth: A workload can be memory-bound yet still fail to use the GPU's maximum theoretical bandwidth. Factors like software overhead, memory access patterns, and kernel-launch latency create a gap between the advertised peak and real-world achieved performance (Source).
What is GPU Memory Bandwidth?
GPU memory bandwidth is the rate at which a GPU processor can read data from or write data to its dedicated memory. It is measured in gigabytes per second (GB/s) or terabytes per second (TB/s). This is distinct from memory capacity, which is the total amount of data the memory can store (measured in GB). Think of it like a water pipe: capacity is the size of the tank, while bandwidth is the width of the pipe determining how fast water can flow.
Modern high-performance GPUs use High Bandwidth Memory (HBM), a high-throughput DRAM technology where multiple memory dies are stacked and connected to the GPU through a very wide interface. This architecture enables high data transfer rates.
A GPU accesses data from a memory hierarchy. The fastest data access is to on-chip registers and caches. The main pool of GPU memory is typically HBM, which is placed close to the GPU. Beyond the GPU's own memory, it can also access the computer's main system (or host) memory, but this access is generally much slower and can become a bottleneck when large amounts of data must be moved between the host and the GPU. This article focuses primarily on the bandwidth between the GPU and its HBM.
The massive parallelism of GPUs, which is why they are so effective for AI, requires an enormous amount of data. HBM bandwidth ensures the processing cores are fed quickly enough to stay busy.
How GPU Memory Works: From HBM to Cores
To understand why bandwidth is a bottleneck, it helps to follow the path data takes inside a GPU. A GPU's memory is a hierarchy of different storage types, each with different speeds and sizes.
- Off-Chip HBM: This is the largest pool of memory, holding the entire model, datasets, and other necessary information like the KV cache (Source). It provides high aggregate bandwidth but is slower than the memory located directly on the processor chip.
- On-Chip Caches (L2/L1): As data is requested from HBM, it is loaded into on-chip caches. These are smaller, much faster pools of memory located closer to the processing cores. They store recently used data to avoid the "trip" back to HBM.
- Shared Memory & Registers: The fastest and smallest memory types are located directly within the GPU's execution units. Registers hold the specific data an instruction is currently operating on.
When a program, or "kernel," runs on the GPU, its threads fetch data from this hierarchy. An operation might require loading model weights from HBM, through the caches, and into registers. After a calculation, the result is written back out through the same path. Every byte that travels from the off-chip HBM to the cores contributes to the bandwidth "cost" of the operation. If a program needs to move a lot of data for very little computation, the system will spend most of its time waiting for data to arrive.
Why is LLM Decoding Memory-Bound?
LLM decoding is the process of generating a response token by token, and it is a classic example of a memory-bound workload. This happens because for every single token generated, the GPU must read a large portion of the model's parameters (weights) from its HBM. This process has very low "arithmetic intensity"—a key concept for understanding performance.
Arithmetic intensity is the ratio of mathematical operations performed to the amount of data moved from memory.
- High Arithmetic Intensity: Performing many calculations for each byte of data loaded. Example: The prefill stage of LLM inference, where a large prompt is processed in a batch, allowing weights to be reused.
- Low Arithmetic Intensity: Performing few calculations for each byte of data loaded. Example: LLM decoding.
During autoregressive decoding, the model generates the next token based on all previous tokens. At each step, this involves a series of matrix-vector operations. The GPU loads huge matrices (the model weights) from HBM to multiply them by a small vector (the current state). Because the vector is small, the GPU performs relatively few calculations before it needs to load the next set of weights for the next token (Source). The powerful tensor cores, designed for massive matrix-matrix math, can remain underutilized while waiting for data. This makes the overall speed dependent on how fast the weights can be read from memory—the GPU memory bandwidth.
This contrasts with the "prefill" stage, where the model processes the user's initial prompt. The prompt can be processed as a larger batch, allowing for more efficient matrix-matrix operations with higher arithmetic intensity. This is why prefill can be compute-bound, while decoding is memory-bound (Source).
How to Know if a Workload is Memory-Bound: The Roofline Model Explained
The Roofline model is a visual tool that plots a system's performance against a workload's arithmetic intensity (Source). It provides a conceptual model for understanding performance bottlenecks. The "roofline" itself is composed of two limits that cap the maximum achievable performance.
The chart's axes are:
- Y-axis: Performance, measured in floating-point operations per second (FLOP/s).
- X-axis: Arithmetic Intensity (AI), measured in FLOPs per byte (FLOPs/byte).
The two "roofs" are:
- The Compute Roof (Horizontal Line): This represents the GPU's peak theoretical compute performance (its TFLOP/s rating). No matter how much data is reused, you cannot perform calculations faster than this limit.
- The Memory Bandwidth Roof (Diagonal Line): This represents the limit imposed by memory bandwidth. The performance on this slope is calculated as
Peak Memory Bandwidth × Arithmetic Intensity.
A workload's position on the chart indicates what is limiting its performance.
- If its arithmetic intensity is low, it falls under the diagonal memory roof. It is memory-bound. To improve performance, one must either increase memory bandwidth or increase the workload's arithmetic intensity (by changing the algorithm).
- If its arithmetic intensity is high, it hits the horizontal compute roof. It is compute-bound. To improve performance here, one would need a GPU with a higher TFLOP/s rating.
The point where the two lines intersect is called the ridge point. This represents the minimum arithmetic intensity a workload needs to become compute-bound on that specific hardware.
A Worked Example: Arithmetic Intensity on a Roofline Chart
Let's make this concrete with a simple, illustrative example of a memory-bound operation: vector addition. Imagine we are computing z[i] = x[i] + y[i] for millions of elements in three arrays.
For each element i, the GPU must perform:
- One floating-point operation: A single addition.
- Three memory transfers:
- Read one element from array
x. - Read one element from array
y. - Write one element to array
z.
- Read one element from array
Let's assume we are using 32-bit floating-point numbers (FP32), which are 4 bytes each.
- Total Operations: 1 FLOP
- Total Bytes Moved: (4 bytes read) + (4 bytes read) + (4 bytes write) = 12 bytes
- Arithmetic Intensity (AI): 1 FLOP / 12 bytes ≈ 0.083 FLOPs/byte
Now, let's place this on a Roofline chart for a hypothetical GPU with a peak compute of 50 TFLOP/s and a peak memory bandwidth of 1.5 TB/s. The ridge point for this GPU would be roughly (50 * 10^12 FLOP/s) / (1.5 * 10^12 B/s) ≈ 33.3 FLOPs/byte.
Our vector addition workload, with an AI of 0.083, falls far to the left of the ridge point. It is deep in the memory-bound region. Its performance is not limited by the GPU's 50 TFLOP/s of compute power. Instead, its performance is limited by the memory bandwidth roof: 1.5 TB/s * 0.083 FLOPs/byte ≈ 0.125 TFLOP/s. The GPU's powerful arithmetic units are mostly idle, waiting for data.
Memory-Bound vs. Compute-Bound Workloads
Different AI tasks have different characteristics. Understanding where common workloads fall can help in choosing the right hardware and optimizing software.
| Workload | Primary Bottleneck | Typical Arithmetic Intensity | Why? |
|---|---|---|---|
| LLM Decoding | Memory-Bound | Low | Repetitive weight/KV cache loading for single-token matrix-vector operations (Source). |
| LLM Prefill | Compute-Bound | High | Large batch processing of input prompts enables efficient matrix-matrix operations with data reuse (Source). |
Common Mistakes and Limits to Know
The Roofline model is a powerful guide, but it simplifies reality. When analyzing performance, it's crucial to avoid common pitfalls.
Mistake 1: Confusing Peak and Achieved Bandwidth
A workload being "memory-bound" does not mean it achieves 100% of the GPU's theoretical peak bandwidth. In practice, achieved bandwidth is often lower. Several factors contribute to this gap:
- Kernel Launch Overhead: The time it takes for the CPU to tell the GPU to start a new task can be significant, especially for very short tasks like generating a single token (Source).
- Inefficient Access Patterns: GPUs read memory in wide chunks. If a kernel requests small, non-contiguous pieces of data, it can lead to wasted bandwidth.
- Software Inefficiencies: The code itself, the compiler, and the runtime libraries can all introduce overhead that prevents the hardware from being fully saturated.
One study found that for a specific LLM, the achieved performance was only about 27% of what the memory bandwidth should have allowed on one high-end GPU, highlighting this gap between theory and reality (Source).
Mistake 2: Assuming Higher Bandwidth is a Silver Bullet
While increasing memory bandwidth helps memory-bound workloads, it does not guarantee a proportional speedup. If performance is limited by launch overhead or other latencies, simply widening the memory "pipe" might not help, because the GPU is spending its time on other things besides moving data. This is particularly true for generating text at "batch one," where the work for each token is very small (Source).
Mistake 3: Focusing Only on TFLOP/s for Inference
A GPU's TFLOP/s rating is a measure of its peak computational throughput. As shown with the Roofline model, this metric can be irrelevant for performance if the workload is memory-bound. For LLM decoding, comparing GPUs based on memory bandwidth is often a much better predictor of real-world performance than comparing their TFLOP/s ratings.
What to learn next
- Model FLOPs Utilization (MFU): Why Your GPU Is Slow - Learn what model flops utilization (MFU) is and why it's a better metric than GPU utilization for training efficiency. Calculate MFU and fix
- GPU Architecture for AI: A Beginner's Guide - Learn how the GPU architecture for AI works. This guide explains how workloads are scheduled on streaming multiprocessors, warps, and CUDA c
- Why GPUs for AI? A Guide to Parallel Computing: Explore the architectural reasons that make GPUs the standard for deep learning and other parallel tasks.
- Reasoning Models Explained: How They 'Think' Before Answering: Learn about advanced AI models that use techniques like chain-of-thought, which have their own unique performance characteristics.
- Automatic Speech Recognition Explained: How It Works: Discover another common AI workload and the computational steps involved in turning audio into text.
