GPUs are expensive, and demand for AI models can be unpredictable. Autoscaling GPU inference helps manage these challenges by automatically adjusting capacity to match traffic. This reduces costs by eliminating idle servers during quiet periods and ensures performance by adding capacity during traffic spikes.
However, autoscaling for GPUs is not like scaling a typical web server. The long startup times and unique performance characteristics of AI models require a more nuanced approach.
Key takeaways
- GPU utilization is often a poor metric for autoscaling. Use leading indicators like request queue depth, complemented by latency guardrails, to make better scaling decisions (Autoscale AI model inference on GPUs with Amazon EKS).
- Autoscaling is a two-step process. A pod autoscaler requests more software replicas, and a separate node provisioner must supply the physical GPU hardware if it's not already available (Autoscale AI model inference on GPUs with Amazon EKS).
- The primary cause of slow scale-up is the "cold start." This delay includes provisioning a new GPU node, downloading the container image, and loading the model into GPU memory (A Faster Cold Start for AI Models).
- Scaling to zero saves money but introduces significant latency for the first request. This tradeoff makes it unsuitable for many latency-sensitive applications (Autoscale AI inference with HPA and KEDA - Amazon EKS).
How Autoscaling GPU Inference Works: A Step-by-Step Breakdown
Autoscaling GPU inference is best understood as a two-stage control system. First, a pod autoscaler decides to change the number of running application copies. Second, if needed, a node autoscaler provisions the underlying hardware. This separation is crucial because new pods can't run if there isn't a free GPU for them.
Here is the full sequence of events when scaling up from a shortage of capacity:
- Metrics Collection: An observability tool like Prometheus collects metrics from the running inference server replicas. These metrics might include the number of requests waiting in a queue, request latency, or GPU utilization.
- Scaling Decision: A pod autoscaler, such as the Kubernetes Horizontal Pod Autoscaler (HPA), compares a metric to its configured target. For example, it might see the average queue depth has exceeded the target of 5 requests. It then decides to increase the desired replica count for the application.
- Pod Scheduling: Kubernetes receives the request for new pods and tries to schedule them onto available nodes in the cluster.
- Pending State: If no existing nodes have an available GPU, the new pods enter a "Pending" state. They cannot start until new hardware is available (Autoscale AI model inference on GPUs with Amazon EKS).
- Node Provisioning: A node autoscaler (like Karpenter or Cluster Autoscaler) detects the Pending pods. It then makes an API call to the cloud provider to provision a new virtual machine with the required GPU.
- Pod Startup: Once the new node joins the cluster, Kubernetes schedules the Pending pod onto it. The node pulls the container image, and the container starts.
- Model Loading: This is a critical part of the "cold start." The inference server application starts up and loads the AI model's weights into the GPU's memory (VRAM). This can involve reading gigabytes of data from storage and copying it to the GPU (A Faster Cold Start for AI Models).
- Ready State: After the model is loaded and the server is fully initialized, the pod's readiness probe passes. Kubernetes marks the pod as "Ready" and begins sending it traffic. During this entire startup process, incoming requests are handled by the original replicas or get queued up.
Why is GPU Autoscaling So Slow? (The Cold Start Problem)
The primary challenge in autoscaling GPU inference is the "cold start" latency. This is the total time from the decision to scale up to the new replica being ready to serve traffic. It is often measured in minutes, not seconds, because it is a sum of several significant delays.
The main components of a cold start are:
- Node Provisioning Delay: A cloud provider can take several minutes to allocate a new GPU-equipped virtual machine and boot its operating system. This is often the longest part of the process.
- Container Image Pull: AI and machine learning container images can be very large, often many gigabytes. Downloading this image to a new node for the first time can be slow.
- Model Loading: This is frequently the biggest application-level delay. The process involves reading the model's weights (tensors) from a storage system into the server's main memory (RAM) and then copying them into the GPU's high-speed VRAM (A Faster Cold Start for AI Models). For a large language model, this can mean moving tens of gigabytes of data.
- Inference Server Initialization: Before serving, the server may need to perform additional setup. This can include allocating a KV cache for generative models, compiling model operations into optimized CUDA kernels, or capturing an execution graph to speed up subsequent requests (A Faster Cold Start for AI Models).
During this entire multi-minute period, your application's capacity is fixed. A sudden traffic spike can lead to long queues or dropped requests while new replicas slowly come online. Understanding and planning for this delay is essential for building a reliable system.
What Metric Should You Use to Autoscale GPU Inference?
Choosing the right metric is the most important decision in designing an autoscaling strategy. A poor metric will cause the system to scale up too late or scale down too aggressively. While GPU utilization seems like an obvious choice, it is often a misleading, lagging indicator of system health (Autoscale AI model inference on GPUs with Amazon EKS).
A better approach uses a leading indicator of demand, like queue depth, combined with a direct measure of user experience, like latency.
| Metric | How it Works | Pros | Cons |
|---|---|---|---|
| GPU Utilization | Scales when the percentage of GPU compute usage crosses a threshold. | Simple to understand and measure. | Poor choice. A lagging indicator. Latency may already be high before utilization climbs. Low utilization doesn't mean the system is idle; it could be bottlenecked elsewhere (Autoscale AI model inference on GPUs with Amazon EKS). |
| Request Queue Depth | Scales when the number of requests waiting to be processed by a replica exceeds a target. | Best leading indicator. A growing queue is the most direct signal that demand is outpacing supply. It triggers scaling before users see high latency (Autoscale AI model inference on GPUs with Amazon EKS). | Can be hard to tune if requests have highly variable processing times. Doesn't directly measure user experience. |
| Latency / TTFT | Scales when average or percentile latency (or Time to First Token for LLMs) exceeds a target. | Essential guardrail. Directly measures the user's experience and protects your Service Level Objectives (SLOs) (Autoscale AI inference with HPA and KEDA - Amazon EKS). | A lagging indicator. By the time latency spikes, users are already impacted. Scaling is reactive, not proactive. |
The best practice is to use a combination of metrics. Use request queue depth as the primary driver for scaling up. Add a latency-based metric as a secondary safety net to ensure your SLOs are always protected.
Kubernetes GPU Scaling: Comparing HPA and KEDA
For teams using Kubernetes, the two most common tools for pod autoscaling are the built-in Horizontal Pod Autoscaler (HPA) and the open-source add-on KEDA (Kubernetes Event-driven Autoscaling).
| Feature | Horizontal Pod Autoscaler (HPA) | KEDA (Kubernetes Event-driven Autoscaling) |
|---|---|---|
| Integration | Built-in to Kubernetes. | An add-on that extends HPA. Requires installation. |
| Triggers | Scales on CPU/memory and custom metrics (e.g., from Prometheus) (Horizontal Pod Autoscaler). | Supports many trigger sources, including Prometheus, Kafka, RabbitMQ, and cloud provider queues (KEDA Documentation). |
| Scale to Zero | No. Cannot scale a workload down to zero replicas. minReplicas must be 1 or greater. |
Yes. Its primary feature is scaling workloads, including GPUs, from zero to N replicas and back down to zero (Autoscale AI inference with HPA and KEDA - Amazon EKS). |
| Activation | N/A | Supports an activationThreshold, which can prevent scaling up from zero for minor, transient traffic spikes (Autoscale AI inference with HPA and KEDA - Amazon EKS). |
| When to Use | Good for simplicity when you always need at least one replica running to serve traffic with low latency. | Ideal for event-driven or asynchronous jobs, or for workloads that can tolerate a cold-start delay in exchange for maximum cost savings. |
In short, use HPA for its simplicity if you always plan to have at least one warm replica. Choose KEDA when you need to scale based on a wider variety of event sources or want to enable scale-to-zero for cost optimization.
A Worked Example: Sizing an Autoscaled Service
Let's make these concepts concrete. Imagine you need to serve a 7-billion-parameter language model with the following requirements:
- Peak Load: 10 requests per second (RPS).
- Performance SLO: p99 latency must be under 2 seconds.
Here is a simplified process for designing the autoscaling configuration.
Step 1: Benchmark a single replica
First, you need to understand the performance of a single instance of your model server. You run a load test and find that a single replica running on one NVIDIA A10G GPU can handle 2 RPS while keeping p99 latency below the 2-second target. At 3 RPS, the latency target is breached. Techniques like continuous batching are often used to maximize this throughput.
Step 2: Calculate required replicas
With a single replica's capacity known, you can calculate the number of replicas needed to handle peak load.
- Formula:
Replicas = Target RPS / RPS per Replica - Calculation:
10 RPS / 2 RPS per Replica = 5 replicas
You need 5 replicas running simultaneously to handle the expected peak traffic. Your autoscaler's maxReplicas setting should be at least 5.
Step 3: Determine GPU nodes
If each replica requires one A10G GPU and you are using single-GPU nodes, you will need 5 GPU nodes provisioned in your cluster at peak. The node autoscaler is responsible for providing these.
Step 4: Analyze the cold start impact
Now, consider the cold start time. Let's assume testing shows that provisioning a new A10G node and loading the 7B model takes a total of 3 minutes.
Scenario A: Scaling from Zero (
minReplicas: 0) If a sudden traffic spike from 0 to 10 RPS occurs, your system will be completely unavailable for 3 minutes. The first replica takes 3 minutes to become ready, and the system needs all 5 to be ready to handle the full load. This is unacceptable for a user-facing application.Scenario B: Scaling from One (
minReplicas: 1) If you keep one replica warm at all times, the initial 2 RPS of a traffic spike can be handled immediately (Inference Scaling). Requests will begin to queue as traffic exceeds 2 RPS, but the system is not fully down. The autoscaler will start provisioning the other 4 replicas, which will come online over the next few minutes. This is a much better user experience.
This example shows the critical tradeoff between cost and latency. Setting minReplicas greater than zero provides a buffer against traffic spikes at the cost of running idle GPUs. The right number depends on your budget and how much latency your users can tolerate. You can learn more about how to model these expenses in our guide to calculating LLM cost per token.
What are the common mistakes in autoscaling GPU inference?
Designing a robust autoscaling system involves avoiding several common pitfalls.
Mistake: Scaling on GPU utilization.
- Fix: Scale on request queue depth as a leading indicator of demand, and use latency as a safety-net metric to protect your user-facing SLOs.
Mistake: Assuming pod scaling provides immediate capacity.
- Fix: Always account for the multi-minute cold start. Measure your total startup time (node provision + image pull + model load) and incorporate that delay into your capacity planning.
Mistake: Scaling down too aggressively.
- Fix: Use a stabilization window or cooldown period for scaling down. This prevents the system from terminating a replica only to need it again a minute later, which leads to "thrashing" and repeated cold starts.
Mistake: Using scale-to-zero for latency-sensitive applications.
- Fix: If your users expect instant responses, set
minReplicasto 1 or more. This ensures a warm replica is always available to handle the first request in a traffic burst.
- Fix: If your users expect instant responses, set
Mistake: Not setting proper readiness probes.
- Fix: Configure your Kubernetes readiness probe to check that the model is fully loaded into GPU memory and the server can successfully process a health-check inference request. A simple process-level check is not enough.
What to learn next
- How LLM Inference Works: Dive deeper into the technical steps of token generation, from the initial prompt processing to the sampling of output tokens.
- Continuous Batching LLM: Learn about a key serving technique that significantly improves GPU throughput by dynamically batching requests, forming the foundation of modern inference servers.
- LLM Cost Per Token: A Practical Calculation Guide: Understand the factors that drive inference costs and learn how to build a model to estimate your expenses accurately.
