Your AI model is training slowly, but your monitoring dashboard shows very high GPU utilization. This common scenario is confusing because it suggests your hardware is working at full capacity. The problem is that "GPU utilization" is often a misleading metric. To understand real performance, you need to look at Model FLOPs Utilization (MFU).
MFU tells you how much of a GPU's theoretical computing power you are actually using for your model's calculations. It helps you find and fix the hidden bottlenecks that waste time and money.
Key takeaways
- GPU utilization is not the same as efficiency. It only measures the percentage of time a GPU's engines are active, not how much useful work they are doing. A GPU can be near 90% utilized but achieve very low MFU (Source).
- Model FLOPs Utilization (MFU) is a calculated metric. It compares the computational throughput you achieve for your specific model to the GPU's theoretical peak performance, giving a much clearer picture of efficiency (Source).
- Low MFU is often caused by bottlenecks outside of pure computation. These include slow data loading, memory bandwidth limits, inter-GPU communication, or using batch sizes that are too small to saturate the hardware.
- Calculating MFU requires careful accounting. You must match the model's FLOPs formula and numerical precision (e.g., BF16) to the correct peak performance of your specific GPU (Source).
- MFU is most useful for compute-bound tasks like training large models. For memory-bound tasks like single-token inference, MFU is expected to be low, and other metrics like latency are more informative.
What is Model FLOPs Utilization (MFU)?
Model FLOPs Utilization (MFU) is a percentage that shows how effectively your hardware's computational power is being used for your specific AI model. It is the ratio of the model-related calculations your GPU actually performs per second to the maximum number of calculations it could theoretically perform. A higher MFU means you are getting more value from your expensive hardware.
To understand MFU, you first have to understand what standard "GPU utilization" is and why it's not enough.
GPU Utilization vs. MFU
Most monitoring tools report GPU utilization. This metric simply measures the percentage of time a GPU engine was active (Source). It tells you if the GPU is "busy" but not what it's busy with. A GPU can be 100% busy doing inefficient work, like waiting for data from memory or running many tiny calculations that don't take full advantage of its parallel processing power.
MFU, on the other hand, is an analytical metric you calculate, not one you measure directly from a hardware counter (Source). It answers a more important question: "How much of the GPU's peak theoretical compute power am I actually using for my model's forward and backward passes?" It focuses on the useful work, ignoring time spent on other tasks.
Think of it like a sports car. GPU utilization tells you if the engine is running. MFU tells you how close you are to the car's top speed. An engine running in neutral at 8,000 RPM is 100% "utilized," but the car is going nowhere.
Why is My GPU Utilization High but My Training is Slow?
High GPU utilization with slow training means your GPU is active but inefficiently used. The hardware is spending time on tasks other than the core matrix multiplications that train your model. This could mean it is waiting for data, communicating with other GPUs, or executing operations that are limited by memory speed instead of compute power.
Imagine a factory assembly line. "GPU utilization" is like having the lights on and the machines running. But if no raw materials arrive (a data bottleneck), or if workers are waiting for parts from another line (a communication bottleneck), the factory produces very little. MFU is the measure of actual finished products coming off the line. One study found that GPU utilization could reach near 90% while MFU was still far from saturated, showing that activity doesn't always equal efficiency (Source).
The following table breaks down the differences between three key performance metrics.
| Metric | What It Measures | How It's Measured | What It Tells You |
|---|---|---|---|
| GPU Utilization | The percentage of time a GPU engine is active. | Sampled by monitoring tools (e.g., nvidia-smi). |
"Is the GPU busy or idle?" High utilization can hide many types of inefficiency. |
| Model FLOPs Utilization (MFU) | The ratio of achieved model FLOPs per second to the hardware's peak theoretical FLOPs per second. | Calculated from throughput and model parameters. | "How efficiently am I using the GPU's compute for my model?" This is the best measure of training compute efficiency. |
| Hardware FLOPs Utilization (HFU) | The rate of actual floating-point operations performed by the hardware's arithmetic units. | Measured with low-level profiling tools. | "How active are the GPU's math units?" This can be higher than MFU if the GPU performs extra work not counted in the model's FLOPs. |
Understanding these distinctions is key. High GPU utilization is a starting point, but MFU gives you the actionable insights needed to speed up your training. To learn more about how GPUs process these operations, see our guide on GPU architecture for AI.
How MFU Works: From Theory to Throughput
Calculating MFU involves combining information about your model, your training throughput, and your hardware specifications. The formula is a simple division:
MFU = (Achieved Model FLOP/s) / (Peak Theoretical FLOP/s)
Here is a step-by-step guide to finding the numerator and the denominator.
Step 1: Estimate Your Model's FLOPs per Token (Numerator Part 1)
First, you need to know how many floating-point operations (FLOPs) are required to process a single token through your model for one forward and backward pass. This number is estimated based on the model's architecture.
For Transformer models, a common approximation for training FLOPs is:
Model FLOPs per token ≈ 6 * P
Pis the number of non-embedding parameters in the model.- The
6accounts for the multiply-accumulate operation in the forward pass (2 * P) and the larger backward pass (roughly4 * P).
This is an approximation. More precise formulas exist that account for attention heads, sequence length, and other architectural details (Source). Consistency in the formula is key when comparing different training runs of the same model.
Step 2: Measure Your System Throughput (Numerator Part 2)
Next, run your training script and measure its stable throughput. Throughput is the number of tokens your system processes per second. Be sure to measure this after any warm-up phases and exclude time spent on evaluation or checkpointing. Average the measurement over several minutes to get a stable value.
Step 3: Calculate Achieved Model FLOP/s
Now, combine the first two steps to find the total useful computations your system is achieving per second.
Achieved Model FLOP/s = (Model FLOPs per token) * (Tokens per second)
Step 4: Find Your Hardware's Peak FLOP/s (Denominator)
This is a critical step where mistakes are common. You must find the theoretical peak performance of your GPU from its official specification sheet. Pay close attention to two things:
- Numerical Precision: GPUs have different peak speeds for different data types (e.g., FP32, FP16, BF16, INT8). You must use the hardware's peak FLOP/s for the exact precision your training job uses. Using the FP32 peak for a BF16 job will give you a misleadingly low MFU.
- Operation Type: The peak FLOP/s often refers to dense matrix-matrix multiplications, which are accelerated by specialized hardware like NVIDIA's Tensor Cores. This is usually the correct number for deep learning training (Source).
Step 5: Calculate Model FLOPs Utilization (MFU)
Finally, divide your achieved FLOP/s by the hardware's peak FLOP/s.
MFU = (Achieved Model FLOP/s) / (Peak Theoretical FLOP/s)
The result is a percentage that represents your training job's computational efficiency.
Calculating MFU: A Worked Example for a Transformer Model
Let's make this concrete with an illustrative example.
Scenario:
- Model: A 1 billion parameter Transformer (
P = 1e9). We'll assume all parameters are non-embedding for simplicity. - Hardware: A single GPU with a peak theoretical performance of 312 TFLOP/s for BF16 precision (e.g., an NVIDIA A100). (1 TFLOP/s = 1 trillion FLOPs per second).
- Training: The model is being trained using BF16 precision, and after optimization, we measure a stable throughput of 25,000 tokens per second.
Step 1: Estimate Model FLOPs per Token
Using the 6 * P rule of thumb:
P= 1,000,000,000 parametersModel FLOPs per token ≈ 6 * 1,000,000,000 = 6 GFLOPs
Step 2: Calculate Achieved Model FLOP/s
Achieved FLOP/s = (6 GFLOPs/token) * (25,000 tokens/sec)Achieved FLOP/s = 150,000 GFLOPs/sec = 150 TFLOP/s
Step 3: Calculate MFU
Now we compare our achieved throughput to the hardware's peak.
Peak FLOP/s = 312 TFLOP/s(from the GPU spec sheet for BF16)MFU = (Achieved FLOP/s) / (Peak FLOP/s)MFU = (150 TFLOP/s) / (312 TFLOP/s) ≈ 0.48
The calculated MFU is 48%. This indicates that the workload is making substantial use of the hardware's compute capabilities, suggesting it is likely compute-bound rather than limited by other bottlenecks like data loading or memory bandwidth. A very low MFU, such as 5%, would suggest a severe bottleneck exists elsewhere in the system.
What Causes Low MFU and How Do I Fix It? A Profiling Checklist
If your MFU is low, your training is being held back by a bottleneck. Use a profiling tool to investigate your workload and find the root cause. Here is a checklist of common culprits.
1. Is your data pipeline the bottleneck?
Your GPU may be sitting idle while it waits for the CPU to load and preprocess the next batch of data. This is known as input pipeline starvation (Source).
- How to check: Use a profiler to view the CPU and GPU timelines. Look for large gaps between training steps on the GPU timeline. Monitor CPU usage; if it's consistently at 100% during training, it's likely a bottleneck.
- How to fix:
- Use more dataloader workers in frameworks like PyTorch.
- Preprocess and save your data to a fast format before training begins.
- Ensure your data is on a fast local disk (like an NVMe SSD) rather than a slow network drive.
2. Are your batch or sequence sizes too small?
Modern GPUs are massively parallel devices. They achieve peak performance when performing large matrix multiplications. Small batch sizes or short sequences lead to small matrix operations that fail to saturate the compute units.
- How to check: Experiment by doubling your batch size and observing the change in step time and MFU. If the step time does not double, your workload was not compute-saturated.
- How to fix:
- Increase the batch size to the largest value that fits in GPU memory.
- If you can't increase the batch size due to memory constraints, use gradient accumulation. This technique processes several smaller "micro-batches" before updating the model weights, simulating a larger effective batch size.
3. Are you memory-bandwidth bound?
Some operations, like the attention mechanism in Transformers, can be limited by how fast the GPU can move data between its memory and its compute cores. This is a common issue and is why understanding GPU memory bandwidth is so important.
- How to check: Profilers can show you memory throughput and the execution time of individual GPU operations (kernels). If you see long-running kernels that have a low compute-to-memory-access ratio, you may be memory-bound.
- How to fix:
- Use optimized kernels like FlashAttention, which are designed to minimize memory traffic.
- Use mixed-precision training (like BF16 or FP16), as smaller data types reduce memory usage and bandwidth requirements.
4. Is communication overhead killing performance?
In multi-GPU or multi-node training, GPUs must communicate to synchronize gradients. This communication can become a significant bottleneck, especially at a large scale (Source).
- How to check: Profilers designed for distributed training can visualize communication operations like
AllReduce. If you see GPUs waiting for long periods during these operations, communication is your bottleneck. - How to fix:
- Ensure you have a fast interconnect between GPUs (e.g., NVLink) and nodes (e.g., InfiniBand).
- Experiment with different distributed training strategies (e.g., data parallelism vs. tensor parallelism).
- Consider using libraries that optimize communication patterns.
When is MFU a Misleading Metric?
While powerful, MFU is not a universal measure of performance. It is most valuable for large, compute-bound training jobs. In some scenarios, a low MFU is expected and does not indicate a problem.
Memory-Bound Workloads The classic example is autoregressive decoding for LLM inference, where the model generates one token at a time. This process involves loading all the model weights from memory just to perform a relatively small amount of computation for the next token. Performance is almost entirely limited by GPU memory bandwidth, not compute. In this case, MFU will be very low (often <5%), and this is normal. Better metrics for this workload are latency (time to generate a token) or throughput (tokens per second).
Comparing Different Setups MFU is not an absolute number for comparing different systems or models. Its value is highly dependent on the FLOP counting convention used, the model architecture, the specific hardware, and the software stack (Source). You cannot fairly compare an MFU of 50% from one research paper to 60% in another without ensuring all these conditions are identical. Some studies even report MFU for specific parts of a model, like attention MFU, which is not comparable to whole-model MFU (Source).
Use MFU as a tool to optimize your workload on your hardware. Track it as you make changes to see if you are improving your system's efficiency.
What to learn next
- NVLink vs InfiniBand: GPU Interconnects for AI Explained - Explore the nvlink vs infiniband comparison for AI workloads. Learn their complementary roles, why NVLink scales up GPUs in a server, and In
- Why GPUs for AI? A Guide to Parallel Computing: Understand the fundamental reasons why GPUs are the workhorse of modern AI.
- GPU Architecture for AI: A Beginner's Guide: Take a closer look at the components inside a GPU that make high-performance computing possible.
- LoRA Fine-Tuning: A Complete Guide to Low-Rank Adaptation: Learn about efficient fine-tuning methods that have different performance characteristics than full training. ===BODY===
