Continuous batching is a serving technique that can significantly increase the throughput of Large Language Models (LLMs). By continuously adding new requests to a running batch as old ones finish, it keeps expensive GPU hardware busy and minimizes wasted computation. This approach is key to building efficient and scalable AI applications.
Key takeaways
- Continuous batching, also known as iteration-level scheduling, increases LLM serving throughput by rebuilding the active batch of requests at each generation step.
- It avoids the major drawback of static batching: GPU idle time caused by short requests waiting for the longest request in a batch to finish.
- This efficiency gain involves a trade-off between throughput, time-to-first-token (TTFT), and completion latency, which is managed by a scheduler (Source).
- Effective continuous batching depends on efficient management of the KV cache. Memory capacity can be a more significant bottleneck than compute power (Source).
- The terms "continuous," "in-flight," and "dynamic" batching are related but have nuanced differences in their implementation.
What Is Continuous Batching and Why Does It Matter for LLMs?
Continuous batching is a scheduling strategy that dynamically adjusts the composition of a batch during LLM inference. Instead of locking a group of requests together until all are complete, it removes finished requests and adds new ones from a queue at each step of the token generation process. This method is also called iteration-level scheduling or in-flight batching (Source).
This technique is essential because of how LLMs generate text. The process, called autoregressive decoding, creates one token at a time. A request for a short answer might finish in a few steps, while a request for a long article could take hundreds. In typical real-world usage, request lengths are varied and unpredictable (Source).
Traditional static batching groups requests and processes them together. The entire batch must wait until the longest request finishes. During this time, the GPU slots for the already-completed shorter requests sit idle, wasting valuable compute resources.
Continuous batching solves this problem by ensuring the GPU is almost always processing active requests. By continuously swapping finished jobs for new ones, it maximizes throughput (the number of tokens generated per second) and overall system efficiency. One serving system, vLLM with PagedAttention, reported 2–4× higher throughput over compared systems in its evaluation by combining continuous batching with efficient memory management, though these gains should not be attributed to continuous batching alone (Source).
How Does Continuous Batching Work? A Step-by-Step Guide
A system using continuous batching revolves around a scheduler that constantly juggles requests to maximize GPU utilization. The process involves a repeating cycle of admitting, processing, and removing requests from an active batch.
Here is a breakdown of the process:
Queueing: When users send prompts to the model, their requests are placed in a waiting queue.
Admission and Prefill: The scheduler selects one or more requests from the queue based on its policy and available resources like GPU memory. For each newly admitted request, the system performs a prefill step, processing the input prompt to generate the first token and initialize the request's KV cache.
Iterative Decoding: The system performs a single decode step for all requests currently in the active batch. This single step generates one new token for every request in the batch.
Completion and Removal: After a decode step, the scheduler checks for finished requests (e.g., those that hit an end-of-sequence token or a length limit). Completed requests are removed from the active batch, and their GPU memory is released.
Joining the Active Batch: At a generation boundary (between decode steps), the scheduler can admit new requests from the queue if resources are free. These new requests, after their own prefill step, join the existing requests that are still generating tokens (Source).
Repeat: The system repeats the cycle from Step 3, continuously performing decode steps for the active batch, removing completed requests, and adding new ones.
Static vs. Continuous Batching: A Worked Example
A quantitative example clearly shows why continuous batching is more efficient. Imagine three requests arrive at an LLM server at the same time.
- Request A: Needs to generate 10 tokens.
- Request B: Needs to generate 50 tokens.
- Request C: Needs to generate 5 tokens.
For this example, assume each decoding step takes 1 unit of time.
Static Batching Timeline
With static batching, all three requests are bundled together and processed as a single unit.
- The batch size is 3.
- The entire batch must run until the longest request (Request B) is finished. This will take 50 decoding steps.
- At step 5: Request C finishes. However, its slot in the batch remains occupied but idle for the next 45 steps.
- At step 10: Request A finishes. Its slot remains idle for the next 40 steps.
- At step 50: Request B finishes, and the entire batch is complete.
Result:
- Total Time: 50 steps.
- Total Useful Work: 5 (C) + 10 (A) + 50 (B) = 65 token-steps.
- Total Capacity: 3 slots × 50 steps = 150 token-steps.
- Wasted Capacity: 150 - 65 = 85 token-steps (57% of capacity wasted).
The GPU was underutilized for most of the process because it was forced to process padding for completed requests.
Continuous Batching Timeline
With continuous batching, the system re-evaluates the batch at each step.
- Steps 1-5: Requests A, B, and C are processed. The batch size is 3.
- At step 5: Request C finishes and is removed. The scheduler immediately admits a new request from the queue, Request D (needs 20 tokens).
- Steps 6-10: Requests A, B, and D are processed. The batch size remains 3.
- At step 10: Request A finishes. The scheduler admits another new request, Request E (needs 30 tokens).
- Steps 11-25: Requests B, D, and E are processed.
- At step 25: Request D finishes. Another request could be admitted if available.
- ...and so on.
Let's compare the time to process just the first three requests (A, B, C):
- Request C finishes at step 5.
- Request A finishes at step 10.
- Request B finishes at step 50.
Result: While the total time for Request B is unchanged, the key difference is what happens in the background. In the 50 steps it took to process A, B, and C, the system also fully processed Request D and made significant progress on Request E.
- Total Time to clear the queue: Significantly lower.
- Total Useful Work: The GPU remains near 100% utilization, processing 3 requests at almost every step.
- Wasted Capacity: Close to zero.
This simple example illustrates the core value of LLM serving with continuous batching: it transforms idle time into productive work, increasing system throughput.
What's the Difference Between Continuous, In-Flight, and Dynamic Batching?
These terms are often used interchangeably, but they can have slightly different meanings depending on the context. Continuous batching is a specific, advanced form of dynamic batching.
Here’s a table to clarify the distinctions:
| Technique | Mechanism | Primary Goal |
|---|---|---|
| Static Batching | A fixed group of requests is processed together. The batch only finishes when the longest request is complete. | Simplicity of implementation. |
| Dynamic Batching | A broad term for strategies that adjust batch size or composition based on system load, latency targets, or queue length. This can happen between full batches. | Improve utilization over static batching. |
| Continuous / In-Flight Batching | A specific type of dynamic batching where the batch is modified during the generation process. Requests are added and removed on an iteration-by-iteration basis (Source). | Minimize idle time within a batch to maximize throughput and GPU utilization. |
In essence, all continuous batching is dynamic, but not all dynamic batching is continuous. Some simpler dynamic batching systems might only decide to form a new batch of a different size after the previous one has fully completed. Continuous batching is the most sophisticated approach, offering the highest potential for throughput by never allowing the GPU to go idle if there are requests in the queue.
What Are the Trade-offs Between Throughput and Latency?
While continuous batching excels at maximizing throughput, it is not a "free lunch." The scheduler's decisions create a complex balance between overall system speed and the performance of individual requests. Maximizing throughput does not guarantee lower latency for every single user (Source).
Time to First Token (TTFT)
TTFT measures how quickly a user sees the first piece of output after sending a prompt. With continuous batching, if the server is already running a full batch of requests, a new incoming request must wait in the queue. A scheduler prioritizing maximum throughput will keep the system fully loaded, which can increase the queueing time for new requests and lead to a higher TTFT.
Completion Latency
Completion latency is the total time from when a request is sent to when the final token is received. Continuous batching can reduce the average completion latency by making the whole system more efficient. However, it can increase tail latency (e.g., the 99th percentile latency). A simple first-in, first-out (FIFO) scheduler might force a short, quick request to wait behind a long-running batch.
Fairness
A scheduler focused solely on packing as many requests as possible onto the GPU can lead to starvation. For example, if the queue is constantly filling with long, complex requests that keep the batch full, a short request might be perpetually postponed. Ensuring fairness requires more sophisticated scheduling policies.
Because of these complex trade-offs, evaluating a serving system requires a holistic view. Reporting only aggregate throughput (tokens/second) is insufficient. A thorough benchmark must also report a range of latency metrics, including time to first token (TTFT), inter-token latency, and final completion latency. Capturing tail latency (e.g., p95/p99) is also critical, as is measuring queue time and request rejection rates. A policy can improve one metric while harming another, so comprehensive reporting is essential for understanding its true performance (Source).
When Not to Use Continuous Batching
Despite its advantages, continuous batching is not a universal solution. Its benefits are minimal or it may be unnecessarily complex in certain scenarios:
- Homogeneous Workloads: If all your requests have very similar input and output lengths, there is little opportunity for optimization. In a static batch where every request finishes at the same time, there is no idle padding to eliminate.
- Extremely Low Latency Requirements: For applications where minimizing the latency of every single request is the absolute top priority (e.g., real-time systems), running a large, continuous batch might introduce unacceptable queueing delays. In such cases, processing requests individually or in very small batches might be preferable, despite the lower throughput.
- Simple, Low-Traffic Applications: Implementing and tuning a continuous batching system adds complexity. For a personal project or a service with very low concurrent user traffic, the overhead may not be justified. A simple static batching or even a one-request-at-a-time approach might be sufficient.
The core benefit of continuous batching comes from exploiting the variability in request lengths to improve efficiency. If that variability doesn't exist, or if throughput is not the primary goal, simpler serving strategies can be more practical.
What to learn next
- LLM Gateway: A Guide to Smart Model Routing - A complete guide to using an LLM gateway. Learn how model routing works, how to route by cost and difficulty, and the common pitfalls to avo
- LLM Cost Per Token: A Practical Calculation Guide - Learn to calculate the true LLM cost per token for APIs and self-hosted models. This guide provides step-by-step formulas for inference cost
- Speculative Decoding: How It Works to Speed Up LLMs - Learn what speculative decoding is and how it works step-by-step. This guide explains how it speeds up Large Language Models without changin
- Dive deeper into the token-by-token generation process with our guide on How LLM Inference Works: From Prompt to Token Stream.
- Understand the critical role of memory in serving performance by reading about The KV Cache LLM Explained: A Complete Technical Guide.
Sources and further reading
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
- Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
- Multi-Bin Batching for Increasing LLM Inference Throughput
- In-Flight Batching
- In-Flight Batching (V0)
- Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching
