Mixed precision training is a technique that accelerates model training and reduces memory requirements by using a combination of different numerical precisions. Instead of performing every calculation in standard 32-bit floating-point (FP32) precision, it selectively uses faster 16-bit formats (FP16 or BF16) for a majority of operations while retaining FP32 for steps that require high precision to maintain model accuracy.
This approach allows you to train larger models or use bigger batch sizes on the same hardware. Modern deep learning frameworks have automated this process, making it accessible with just a few lines of code.
Key takeaways
- Faster Training: Mixed precision uses 16-bit formats (FP16, BF16) that run much faster on modern GPUs with specialized hardware like Tensor Cores.
- Less Memory: 16-bit numbers use half the memory of 32-bit numbers, allowing for larger models or bigger batch sizes on the same hardware.
- Maintains Accuracy: It keeps a master copy of weights in 32-bit precision (FP32) and performs sensitive operations like loss calculation in FP32 to prevent numerical errors.
- Loss Scaling is Key for FP16: FP16 has a narrow numerical range, so a technique called loss scaling is needed to prevent small gradients from disappearing (underflow).
- BF16 is Often Simpler: Bfloat16 (BF16) has the same wide range as FP32, so it usually does not need loss scaling, making it simpler to use on supported hardware.
What is mixed precision training?
Mixed precision training is a strategy that uses lower-precision numbers for most of a model's calculations to improve speed and save memory. It intelligently combines 16-bit and 32-bit floating-point formats, using the faster, less precise format where it is safe and the slower, more precise format where it is necessary for numerical stability (Source name).
This is not the same as naively converting an entire model to a 16-bit format, which can cause training to fail. Instead, it is a selective process.
The main goals are:
- Increase Speed: Modern GPUs have specialized hardware, like Tensor Cores, that perform matrix calculations much faster in 16-bit precision than in 32-bit.
- Reduce Memory: 16-bit numbers take up half the memory of 32-bit numbers. This reduces the memory footprint of the model's weights, activations, and gradients, allowing for larger models or bigger batch sizes.
A key part of this strategy is maintaining a full-precision (FP32) copy of the model's weights. During each training step, a 16-bit version of the weights is used for the fast forward and backward passes. However, the small gradient updates are applied to the master FP32 copy. This prevents tiny updates from being rounded to zero, which would stall the learning process (Source name).
Frameworks provide "Automatic Mixed Precision" (AMP) features that handle this automatically. An autocast manager determines which operations can safely run in 16-bit and which should remain in 32-bit (Source name).
How does mixed precision work step-by-step?
To understand mixed precision training, let's walk through a single iteration of a training loop, focusing on the FP16 case which includes loss scaling. The process ensures that fast, low-precision hardware is used for heavy lifting (forward and backward passes), while high-precision storage and updates protect the model from numerical errors.
- Start with FP32 Master Weights. The optimizer holds the authoritative, high-precision (FP32) version of the model's parameters.
- Cast Weights for Computation. Before the forward pass, a low-precision (FP16 or BF16) copy of the weights is created. This is the version that will be used for the matrix multiplications.
- Run Autocast Forward Pass. The model executes its forward pass inside an
autocastcontext. This manager automatically directs operations to run in their optimal precision. For example, convolutions and linear layers run in FP16/BF16, while numerically sensitive operations like layer normalization or softmax might be kept in FP32 for stability (Source name). Activations generated during this pass are also stored in low precision, saving memory. - Calculate Loss in FP32. The final output of the model is typically cast back to FP32 before the loss is calculated. This ensures the loss value is precise, which is important for calculating accurate gradients.
- Scale the Loss (FP16 only). If using FP16, the loss value is multiplied by a large number called a scale factor (e.g., 65,536). This artificially inflates the loss, which in turn inflates the gradients during backpropagation. This step is not needed for BF16 (Source name).
- Run Backward Pass. The backward pass calculates gradients with respect to the scaled loss. Because the loss was scaled up, the resulting gradients are much larger and less likely to "underflow" (become zero) in the limited range of FP16.
- Unscale Gradients and Check for Overflow. Before the optimizer uses the gradients, they are unscaled by dividing them by the same scale factor. The framework also checks if any gradients have become
inf(infinity) orNaN(Not a Number). If overflow is detected, the entire update for this step is skipped to prevent corrupting the master weights. - Update FP32 Master Weights. If the gradients are valid, the optimizer uses these unscaled, FP32 gradients to update the master FP32 weights. The cycle then repeats.
Why is BF16 better than FP16 for training?
Bfloat16 (BF16) is generally preferred over float16 (FP16) for training deep learning models on hardware that supports it. The key difference lies in how they allocate their 16 bits between representing the range (exponent) and the precision (mantissa) of a number. This makes BF16 more robust against certain numerical errors (Source name).
A floating-point number is like scientific notation: it has a sign, a significand (the digits), and an exponent (where the decimal point goes).
- BF16 (bfloat16): Prioritizes range. It uses 8 bits for the exponent, the same as the 32-bit FP32 format. This gives it a massive numerical range, making it very unlikely that a number will be too large (overflow) or too small (underflow). However, it only uses 7 bits for precision.
- FP16 (float16): Prioritizes precision. It uses 10 bits for the significand, offering more precision than BF16. However, it only has 5 bits for the exponent, giving it a very narrow numerical range. This makes it vulnerable to underflow, where small gradients become zero.
For most deep learning training, BF16's wide range means you usually do not need loss scaling. This simplifies the training loop and removes a potential source of instability, making it the easier and often more reliable choice on compatible hardware (Source name).
| Feature | FP16 (float16) | BF16 (bfloat16) | FP32 (float32) - Reference |
|---|---|---|---|
| Total Bits | 16 | 16 | 32 |
| Exponent Bits | 5 | 8 | 8 |
| Mantissa/Significand Bits | 10 | 7 | 23 |
| Dynamic Range | Narrow | Wide (Same as FP32) | Wide |
| Precision | Higher (than BF16) | Lower (than FP16) | Highest |
| Requires Loss Scaling | Yes, almost always | No, typically not needed | No |
Why is loss scaling crucial for FP16?
Loss scaling prevents small gradients from vanishing when using the FP16 data type, a problem known as underflow. FP16 has a limited range and cannot represent very small positive numbers. Loss scaling moves these small gradients into a range that FP16 can handle, ensuring the model continues to learn from subtle signals (Source name).
Let's see this with an illustrative example.
The Scenario
Imagine during backpropagation, our model calculates a tiny but important gradient value: 0.00001.
The FP16 Problem: Underflow
The smallest positive number that standard FP16 can represent is approximately 0.00006. Our calculated gradient of 0.00001 is smaller than this minimum. When we try to store it in an FP16 tensor, it gets rounded down to zero.
- Original Gradient:
0.00001 - Gradient in FP16:
0.0(Underflow!)
The weight update is lost. The model fails to learn from this training example.
The Loss Scaling Solution
- Scale Up Before Backward Pass: We choose a large scaling factor, for instance,
65,536(2^16), and multiply our loss by it before starting backpropagation. - Calculate Scaled Gradient: Because of the chain rule in calculus, all gradients calculated during the backward pass will also be scaled by this factor.
- Scaled Gradient:
0.00001 * 65,536 = 0.65536
- Scaled Gradient:
- Represent in FP16: The value
0.65536is well within the range of FP16 and can be stored without being rounded to zero. The information is preserved. - Scale Down Before Optimizer Step: After the backward pass is complete and before the optimizer updates the weights, we divide all gradients by the scaling factor to restore their original values.
- Unscaled Gradient:
0.65536 / 65,536 = 0.00001
- Unscaled Gradient:
Result: The optimizer receives the correct, small gradient of 0.00001. It can now make the subtle adjustment to the model's weights, allowing learning to proceed correctly.
How do you implement mixed precision in code?
Modern frameworks like PyTorch and TensorFlow make it simple to enable Automatic Mixed Precision (AMP). You typically only need to add a few lines of code to your existing training loop.
PyTorch
In PyTorch, you use torch.autocast and torch.cuda.amp.GradScaler. The autocast context manager automatically casts operations to lower precision, while GradScaler handles the loss scaling and unscaling needed for FP16 stability (Source name).
# Minimal PyTorch example for mixed precision with FP16
# Create a GradScaler instance once
scaler = torch.cuda.amp.GradScaler()
for data, label in train_loader:
optimizer.zero_grad()
# Enter the autocast context manager for the forward pass
with torch.autocast(device_type="cuda", dtype=torch.float16):
output = model(data)
loss = loss_fn(output, label)
# Scale the loss and call backward() on the scaled loss
scaler.scale(loss).backward()
# scaler.step() unscales gradients and calls optimizer.step()
scaler.step(optimizer)
# Update the scale for next iteration
scaler.update()If you were using BF16, which is supported on newer hardware, you would change dtype to torch.bfloat16 and could remove the GradScaler, as loss scaling is not needed.
TensorFlow
In TensorFlow with Keras, enabling mixed precision is often even simpler. You set a global policy at the beginning of your script. The framework then automatically handles the casting and loss scaling within the standard model.fit() training loop.
To use mixed precision with FP16:
# Enable mixed_float16 policy in TensorFlow
from tensorflow import keras
keras.mixed_precision.set_global_policy('mixed_float16')To use mixed precision with BF16:
# Enable mixed_bfloat16 policy in TensorFlow
keras.mixed_precision.set_global_policy('mixed_bfloat16')TensorFlow's policy automatically wraps the optimizer to apply loss scaling when mixed_float16 is used (Source name).
What are common mistakes when using mixed precision?
While frameworks automate much of the process, several common issues can still arise. Awareness of these pitfalls is key to successful debugging.
- Applying Gradient Clipping Incorrectly: When using loss scaling with FP16, gradients must be unscaled before clipping is applied. Clipping the scaled gradients uses an incorrect threshold, which can harm training stability.
- Ignoring Nonfinite Gradients: Overflow during the backward pass can produce
inforNaNgradients. If the optimizer step for that batch is not skipped, these nonfinite values will corrupt the model's FP32 master weights and can permanently destabilize training. - Assuming BF16 is a Silver Bullet: While BF16's wide exponent range prevents most underflow and overflow issues, its low mantissa precision (fewer significant digits) can still cause problems in algorithms that are sensitive to rounding errors.
- Benchmarking Performance Improperly: Accurate speed comparisons require running several warm-up iterations before timing. Furthermore, GPU operations should be synchronized before measuring time, and performance depends on using hardware-appropriate tensor shapes.
- Not Verifying Hardware Support: Performance gains rely on specific hardware acceleration (e.g., Tensor Cores) for lower-precision formats and the availability of optimized kernels in the deep learning framework for the operations being used.
What to learn next
- Training Checkpointing: A Guide to Fault-Tolerant ML - Learn to use training checkpointing for fault-tolerant ML. Our guide covers strategy, save frequency, and how to resume training after an in
- LLMOps: A Practical Guide to Operating LLM Applications - LLMOps is the engineering discipline for building and operating reliable LLM applications. Learn the complete lifecycle, key components, and
- Tensor Cores Explained: Dive into the specialized GPU hardware that makes mixed precision training so fast.
- Distributed Training LLM: Data, Tensor, and Pipeline Parallelism Explained: Learn how memory savings from mixed precision can be combined with distributed techniques to train even larger models across multiple GPUs.
- Direct-Preference-Optimization (DPO) Explained: Explore advanced techniques for fine-tuning models, where the efficiency gains from mixed precision are highly valuable.
