When a large language model (LLM) is too big for a single GPU, you must split it across multiple devices. This general strategy is called model parallelism, and the two most common methods are tensor parallelism and pipeline parallelism. Understanding the difference is crucial for building efficient training and inference systems.
This guide explains the mechanics of tensor parallelism vs pipeline parallelism, helping you choose the right approach for your workload.
Key takeaways
- Granularity: Tensor parallelism (TP) splits computation within a single layer, like a large matrix multiplication. Pipeline parallelism (PP) splits a model between layers, assigning a sequence of entire layers to each GPU (Source).
- Communication: TP requires frequent communication within each forward and backward pass, making it sensitive to network latency. It works best with ultra-fast interconnects like NVLink. PP communicates larger chunks of data (activations) between stages less frequently, making it more tolerant of slower networks (Source).
- Performance: TP can reduce the latency of a single computation but introduces synchronization overhead. PP excels at maximizing throughput by keeping GPUs busy with a continuous stream of data but adds latency due to its sequential nature and an initial "bubble" of idle time (Source).
- Best of Both: For very large systems, a hybrid approach is common. Tensor parallelism is used within a server node to handle massive layers, while pipeline parallelism is used across nodes to split the model's depth (Source).
When do you need to shard a model?
You must shard a model when it is too large to fit into a single GPU's memory. This memory footprint includes not just the model's weights but also several other components that consume VRAM. Sharding can also be a performance optimization, even if the model fits on one device, by allowing more computations to happen in parallel (Source).
During training, a GPU's memory holds:
- Model Weights: The parameters of the neural network.
- Gradients: The calculated changes to the weights during backpropagation. These are typically the same size as the model weights.
- Optimizer States: Optimizers like Adam store running averages (momentum and variance), which can significantly increase memory requirements.
- Activations: The intermediate outputs of each layer, which are saved during the forward pass to be used in the backward pass.
During inference, the memory load is different but still significant:
- Model Weights: These are required for the forward pass.
- Key-Value (KV) Cache: To generate text efficiently, LLMs cache the keys and values from the attention mechanism for all previous tokens in a sequence. This cache can grow very large, especially with long sequences and large batch sizes.
Sharding a model always introduces communication overhead. The performance gain from parallel execution must outweigh the cost of sending data between GPUs. You can learn more about calculating these memory requirements by reading this guide on choosing a GPU for LLM inference.
How tensor parallelism works: Splitting a single layer
Tensor parallelism divides the math inside a single large layer across multiple GPUs. It works by splitting the model's large weight matrices (tensors) into smaller chunks. Each GPU holds one chunk and performs a piece of the calculation. This method is ideal for models with very wide layers that are themselves too large for one GPU (Source).
Let's walk through an example of a Transformer's MLP block, which contains two linear layers. We'll split it across two GPUs.
Step 1: Column-wise split of the first matrix
The first linear layer performs the matrix multiplication Y = X * A. In tensor parallelism, we split the weight matrix A column-wise into A1 and A2. Each GPU computes part of the result.
- An input tensor
Xis sent to both GPUs. - GPU 1 calculates
Y1 = X * A1. - GPU 2 calculates
Y2 = X * A2.
The full result Y would be [Y1, Y2], but the pieces are kept separate on each GPU for now.
Step 2: Row-wise split of the second matrix
The second linear layer performs Z = Y * B. To make this work with our split Y, we split the second weight matrix B row-wise into B1 and B2.
- GPU 1 receives
Y1(from its previous step) and calculates a partial resultZ1 = Y1 * B1. - GPU 2 receives
Y2and calculates its partial resultZ2 = Y2 * B2.
Note that each GPU only needs a piece of Y and a piece of B to compute its partial Z.
Step 3: Combine results with all-reduce
At this point, GPU 1 has Z1 and GPU 2 has Z2. The correct final output is Z = Z1 + Z2. To get this result, the GPUs perform an all-reduce operation.
An all-reduce is a collective communication operation where each GPU sends its partial result to all other GPUs. The GPUs then sum the results they receive, so every GPU ends up with the identical, final result Z.
This process of splitting matrices and synchronizing results happens within every parallelized layer of the model. A standard Transformer block using this method may require two all-reduce operations in the forward pass and two more in the backward pass during training (Source). This highlights why tensor parallelism is so dependent on fast, low-latency interconnects.
How pipeline parallelism works: A multi-GPU assembly line
Pipeline parallelism splits a model by depth, creating a multi-stage assembly line across GPUs. Instead of splitting individual layers, you assign a whole group of consecutive layers to each GPU. Each group of layers on a GPU is called a "stage." This is particularly effective for very deep models (Source).
For example, you could split a 24-layer model across 4 GPUs, with each GPU holding 6 layers (Source).
The problem: Idle GPUs
A naive approach would be to process one input sequence at a time.
- GPU 1 processes the input through layers 1-6.
- GPU 1 sends its output to GPU 2.
- GPU 2 processes it through layers 7-12, and so on.
In this scenario, only one GPU is active at any given moment. GPUs 2, 3, and 4 are idle waiting for GPU 1 to finish. This is extremely inefficient.
The solution: Micro-batching
To solve this, pipeline parallelism splits a large data batch into smaller micro-batches. This allows the GPUs to work concurrently on different micro-batches, like an assembly line.
Here’s how it works with 4 GPUs (stages) and 4 micro-batches (MB1, MB2, etc.):
- Time 1: GPU 1 processes MB1.
- Time 2: GPU 1 sends its output to GPU 2, which starts on MB1. Simultaneously, GPU 1 starts processing MB2.
- Time 3: GPU 2 sends its output to GPU 3 (for MB1). GPU 1 sends its output to GPU 2 (for MB2). GPU 1 starts on MB3.
- Time 4: All 4 GPUs are finally active. GPU 4 works on MB1, GPU 3 on MB2, GPU 2 on MB3, and GPU 1 on MB4.
This staggered execution allows for much higher hardware utilization, which is key for maximizing throughput. Techniques like continuous batching are often used to keep this pipeline full in a real-world inference server.
The pipeline bubble
The pipeline is not always full. At the beginning, GPUs in later stages are idle. At the end, GPUs in early stages finish first and become idle. This period of underutilization is called the pipeline bubble. The size of the bubble represents the inefficiency of the pipeline.
The bubble's cost is relative. With many micro-batches, the time spent filling and draining the pipeline is small compared to the time it runs at full capacity. However, for workloads with small batches or single requests, the bubble can dominate and lead to poor performance (Source).
Tensor vs. pipeline parallelism: A head-to-head comparison
The choice between tensor and pipeline parallelism depends on your model architecture, hardware, and performance goals. Here is a direct comparison of their key attributes.
| Attribute | Tensor Parallelism | Pipeline Parallelism |
|---|---|---|
| Granularity | Intra-layer (splits large weight matrices) | Inter-layer (splits the model into sequential stages) |
| Communication Pattern | Frequent, small, collective operations (e.g., all-reduce) within each layer | Less frequent, large point-to-point transfers of activations between stages |
| Network Sensitivity | High. Requires low-latency, high-bandwidth interconnects like NVLink. | Moderate. Can tolerate slower inter-node links (e.g., Ethernet), but still benefits from high bandwidth. |
| Latency (Single Request) | Lower, as all GPUs compute a layer simultaneously. Synchronization adds overhead. | Higher, due to sequential processing across stages. |
| Throughput (Batch) | Limited by synchronization overhead. | High, as micro-batching keeps hardware busy and amortizes the pipeline bubble. |
| Memory Savings | Reduces memory for weights, gradients, and optimizer states within each layer. | Divides the total number of layers (and their state) across GPUs. |
| Implementation | High complexity. Requires modifying layer internals and managing collectives. | Moderate complexity. Involves model partitioning and scheduling logic. |
Worked example: Choosing a strategy for a 4-GPU setup
Let's apply these concepts to a practical scenario: serving a large language model on a single server with 4 GPUs connected by a fast NVLink interconnect.
Option 1: 4-way tensor parallelism
Each of the 4 GPUs would hold 1/4 of every large weight matrix in the model.
- Pros: This could offer the lowest possible latency for generating a single token, as all four GPUs work together on each layer. It's a good choice if your top priority is minimizing response time for single-user requests.
- Cons: The communication overhead is very high. Every parallelized layer requires collective communication across all four GPUs. This heavy synchronization can become a bottleneck, limiting overall throughput.
Option 2: 4-way pipeline parallelism
Each GPU would hold 1/4 of the model's layers (e.g., GPU 1 has layers 1-20, GPU 2 has layers 21-40, etc.).
- Pros: Communication is simpler and less frequent, occurring only between adjacent stages (GPU 1 sends to 2, 2 to 3, etc.). This can lead to higher overall throughput when serving a batch of many requests, as the pipeline keeps all GPUs busy.
- Cons: The latency for a single request will be higher than with tensor parallelism because the computation is sequential across the four GPUs. The system is only efficient if you can serve a large enough batch to hide the pipeline bubble.
Option 3: 2-way TP x 2-way PP (Hybrid)
This hybrid approach combines both strategies. We create two pipeline stages. Within each stage, we use 2-way tensor parallelism.
- Setup: GPUs 0 and 1 form Stage 1, holding the first half of the model's layers. GPUs 2 and 3 form Stage 2, holding the second half.
- Execution: Within Stage 1, GPUs 0 and 1 cooperate using tensor parallelism to compute their layers. Once done, the result is sent to Stage 2, where GPUs 2 and 3 take over, also using tensor parallelism.
- Pros: This is often the best of both worlds. It handles layers that are too big for one GPU (via TP) while also splitting the model's depth for better memory distribution and throughput (via PP). The high-frequency TP communication happens over the fast NVLink between a pair of GPUs, while the lower-frequency PP communication happens between the stages.
- Cons: This is more complex to implement and tune than a pure strategy.
Decision: For a general-purpose inference server on this hardware, the hybrid approach (Option 3) is often the most balanced and performant choice. It provides a good compromise between latency and throughput.
When not to use model parallelism
While powerful, model parallelism is not always the answer. Applying it unnecessarily can hurt performance due to communication overhead.
- If the model fits on a single GPU: If your model and its entire working state (including KV cache for your target batch size and sequence length) fit comfortably on one GPU, do not use model parallelism. The communication cost will almost certainly outweigh any benefit (Source). For an overview of how these components contribute to memory usage, see this guide on how LLM inference works.
- When computation is slower than communication: Sharding is not automatically beneficial. If the computational work on each device is not substantial enough to hide the latency of communication and synchronization, the parallelized version can be slower than running on a single device (Source).
- If you have a slow interconnect: Do not attempt high-degree tensor parallelism across GPUs connected by a slow link like PCIe or Ethernet. The frequent all-reduce operations will become a severe bottleneck. Pipeline parallelism is more tolerant of slower links but will still be limited by the bandwidth needed to transfer activations.
Before sharding your model, always consider alternatives like using a larger GPU, quantizing the model to reduce its size, or using techniques that reduce activation or optimizer state memory. Model parallelism should be your tool when these other methods are insufficient.
What to learn next
- On Device LLM: The Complete How-It-Works Guide - Learn how an on device LLM runs locally on your phone or laptop. This guide explains memory needs, GGUF, llama.cpp, and the trade-offs of ed
- Distributed Training LLM: Data, Tensor, and Pipeline Parallelism Explained - A guide to distributed training llm techniques on multiple GPUs. Learn how data, tensor, and pipeline parallelism work, compare FSDP vs. ZeR
- Continuous Batching LLM: How It Works & Boosts Throughput: Learn how modern inference servers keep pipeline parallelism systems full to maximize GPU utilization and throughput.
- Choosing a GPU for LLM Inference: A Sizing Guide: A practical guide to calculating the VRAM you need for inference, which helps you decide if model parallelism is necessary.
- How LLM Inference Works: From Prompt to Token Stream: Understand the mechanics of token generation, including the role of the KV cache, to better grasp memory and computational requirements. ===BODY===
