Training large AI models can take days or weeks on many GPUs. A single hardware failure could wipe out days of progress. Training checkpointing is a technique for making these long runs fault-tolerant. It involves periodically saving a recoverable snapshot of the training state—including model parameters, optimizer state, scheduler state, and data position—allowing you to resume from where you left off after an interruption.
Key takeaways
- A complete checkpoint must include more than just model weights. To resume training faithfully, you need the optimizer state, learning rate scheduler, random number generator state, and your position in the dataset.
- Checkpoint frequency is a critical trade-off. Saving too often adds significant overhead and slows down training, while saving too infrequently risks losing too much work after a failure.
- For large models spread across many GPUs, use checkpoint sharding. This technique has each GPU save its own portion of the state, avoiding memory and I/O bottlenecks.
- It is critical to use atomic write patterns to save checkpoints. This involves writing to a temporary directory and then renaming it, which prevents a failed save from corrupting your last good checkpoint.
- The choice of storage—local disk, a shared filesystem, or cloud object storage—impacts performance, cost, and resilience. Your strategy should match your cluster's specific failure modes.
What Should a Full Training Checkpoint Contain?
To resume a training run without altering its trajectory, you must save the complete computational state, not just the model weights. A common mistake is saving only the model's parameters, which forces the optimizer to start over and disrupts the learning rate schedule. This can hurt model convergence.
A full training checkpointing state must include these components:
- Model State: The parameters (weights and biases) of your model. In PyTorch, this is captured with
model.state_dict(). - Optimizer State: The optimizer's internal data, such as momentum and variance estimates for adaptive optimizers like Adam (
optimizer.state_dict()). This is critical for maintaining the learning trajectory. - Scheduler State: The current state of your learning rate scheduler to ensure the learning rate continues along the correct path after resuming (
scheduler.state_dict()). - Mixed Precision Scaler State: If you are using automatic mixed precision, the gradient scaler's state must be saved to prevent numerical instability (
scaler.state_dict()). Learn more about how this works in our guide to mixed precision training. - Random Number Generator (RNG) State: The state of all RNGs (e.g., for PyTorch, NumPy, and Python's
randommodule). This ensures that data shuffling, dropout, and other stochastic operations are reproducible. - Training Progress: The current epoch and step number. This tells your training loop where to resume and ensures logs and learning rate schedules are correct.
- Data Loader / Sampler State: The internal state of your data sampler. This is vital for resuming from the correct batch in your dataset, especially for massive datasets that are not shuffled every epoch.
- Run Metadata: Any other information needed to recreate the training environment, such as configuration files, code versions, and environment variables.
Saving all these elements ensures that a resumed run is a faithful continuation of the original (NVIDIA NeMo documentation).
How Does Training Checkpointing Work? A Step-by-Step Guide
The process involves capturing the training state, writing it durably to storage, and then using it to recover after an interruption. An effective implementation handles potential failures during the save process itself to avoid corruption. This end-to-end workflow protects your computation.
Here is a step-by-step breakdown of the process:
- Choose a Recovery Point: Select a well-defined point in the training loop to save, such as after an optimizer update, rather than at an arbitrary time. This ensures a consistent state for resumption.
- Capture State: At the designated training step, the framework gathers all necessary components. This includes the model parameters, optimizer state, scheduler state, RNG states, and current step number.
- Serialize and Write: The captured state is converted into a byte stream and written to a storage medium. In distributed training, this is often done in a sharded manner, where each process writes its own data shard.
- Use Atomic Writes: To prevent corruption, avoid writing directly over the previous checkpoint. Instead, write the new checkpoint to a temporary directory or file. Only after all data has been successfully written and flushed to disk, rename the temporary directory to its final name. This ensures that an interruption during the save process doesn't leave you with a partial, unusable checkpoint.
- Validate (Optional but Recommended): After writing, run a quick validation check. This could be as simple as verifying that all expected files exist or as thorough as loading the checkpoint in a separate process to confirm its integrity.
- Manage Retention: To control storage costs, delete old checkpoints according to a defined policy (e.g., keep the last three). Crucially, you should only delete an old checkpoint after the new one has been successfully written and validated.
- Resume from Failure: To recover, the training script loads the latest valid checkpoint. It restores all saved components—model, optimizer, scheduler, RNGs—to their respective devices, advances the data loader to the saved step, and restarts the training loop.
- Monitor and Adjust: Track metrics like checkpoint duration, storage throughput, failed saves, and recovery time. Use these measurements to refine the checkpoint interval and storage design, optimizing the trade-off between overhead and reliability.
How Often Should You Save Checkpoints? A Practical Formula
Choosing the right checkpoint frequency is a balance between the cost of saving and the cost of re-computation after a failure. Saving too often slows down training, while saving too infrequently means you lose more work when a failure occurs. You can find an optimal interval with a simple formula.
The goal is to minimize the total time wasted on both checkpointing and re-computation. The optimal checkpoint interval, I_opt, can be estimated with the following formula:
I_opt = sqrt(2 * T_ckpt * MTBF)
Where:
I_optis the optimal time interval between checkpoints (in seconds).T_ckptis the time it takes to save one checkpoint (in seconds).MTBFis the Mean Time Between Failures for your system (in seconds).
This formula provides a mathematically grounded starting point for your checkpointing strategy (Convergence-aware optimal checkpointing study).
Worked Example
Let's apply this to a real-world scenario to see how it works.
Assumptions:
- You are training a large model on a GPU cluster.
- Time to save one checkpoint (
T_ckpt): You measure that saving a full sharded checkpoint takes 5 minutes (300 seconds). - Mean Time Between Failures (
MTBF): From historical data, you know that a node in your cluster fails roughly once every 5 days. 5 days = 5 * 24 * 60 * 60 = 432,000 seconds.
Calculation:
- Plug the values into the formula:
I_opt = sqrt(2 * 300 * 432,000) - Calculate the product inside the square root:
I_opt = sqrt(259,200,000) - Calculate the square root:
I_opt ≈ 16,100 seconds - Convert to a more practical unit:
16,100 seconds / 3600 seconds/hour ≈ 4.47 hours
Conclusion: Based on this calculation, a good starting point is to save a checkpoint approximately every 4.5 hours. This balances the 5 minutes of overhead per save against the risk of losing, on average, a couple of hours of work when a 5-day MTBF failure occurs. This data-driven approach is a key part of effective GPU cost optimization.
In practice, you would round this to a convenient number of training steps that corresponds to roughly 4.5 hours of training.
Distributed Training and Checkpoint Sharding Explained
When a model is too large to fit on a single GPU, it must be split across multiple devices in a process called distributed training. A naive checkpointing approach—gathering all model parts onto one GPU to save—would fail due to memory limits and network congestion. Checkpoint sharding solves this problem.
In checkpoint sharding, each worker process saves only the portion of the model state it owns directly to storage. Instead of one large checkpoint file, you get a directory containing multiple smaller files (shards), one for each process, plus a metadata file that describes how to piece them back together.
This approach has several advantages:
- Avoids Out-of-Memory Errors: No single device needs to hold the entire model or optimizer state.
- Reduces Network Bottlenecks: Data is written in parallel from each node to storage, avoiding a central collection point.
- Faster Saves: Parallel writes to a capable storage system can be much faster than a single, large serial write.
A sharded checkpoint directory for a job running on four GPUs might look like this:
/checkpoints/my-run/step-5000/
├── rank_0_model.pt
├── rank_0_optimizer.pt
├── rank_1_model.pt
├── rank_1_optimizer.pt
├── rank_2_model.pt
├── rank_2_optimizer.pt
├── rank_3_model.pt
├── rank_3_optimizer.pt
└── metadata.jsonWhen resuming, the training framework reads the metadata file and instructs each rank to load its specific shard. A key consideration is that sharded formats can be tied to the specific hardware and parallelism configuration (NVIDIA Megatron-LM documentation). Resuming on a different number of GPUs often requires a separate conversion step. For more details on parallelism strategies, see our guide on distributed training for LLMs.
Choosing Your Checkpoint Storage
The right storage system for your checkpoints depends on your cluster's architecture, budget, and fault tolerance requirements. Writing very large checkpoints can consume substantial network and storage bandwidth, so this choice has a major impact on training efficiency. Each option presents a different trade-off between performance, cost, and resilience.
Here is a comparison of common storage options:
| Storage Type | Speed | Node Failure Resilience | Rack/Cluster Failure Resilience | Best For |
|---|---|---|---|---|
| Local Node Storage | Very High | None | None | Extremely fast, short-term saves. Often used as a temporary staging area before uploading to durable storage. |
| Shared Network Filesystem | Medium | Good | Depends on architecture | General-purpose use in on-premise clusters where a high-performance shared filesystem is available. |
| Parallel Filesystem | Very High | Good | Good | High-performance computing (HPC) environments where maximum I/O throughput is needed for many nodes. |
| Cloud Object Storage | Low to Medium | Excellent | Excellent | Cloud-based training. Offers the highest durability and scalability but typically has higher latency than filesystems. |
A common advanced strategy is tiered checkpointing. A checkpoint is first saved quickly to local disk. Then, an asynchronous process copies it from the local disk to more durable but slower object storage. This approach minimizes the time training is paused while still ensuring long-term fault tolerance against larger-scale failures, as described in research on asynchronous checkpointing schemes.
Common Checkpointing Pitfalls
- Testing Only the Save, Not the Resume: A common error is confirming that a checkpoint file is written without testing if a job can actually be terminated and resumed from it. The only valid test is a full kill-and-resume cycle that verifies the step number, learning rate, and data position are correct (NVIDIA NeMo documentation).
- Saving Only Model Weights: Resuming from weights alone discards optimizer state, scheduler progress, and RNG state, which can alter the training trajectory and prevent a faithful continuation.
- Overwriting the Last Good Checkpoint: Writing a new checkpoint directly over the previous one is risky. A failure during the write process can corrupt the file, leaving you with no valid checkpoint to recover from.
- Ignoring Distributed System Complexities: Assuming a checkpoint created with one number of GPUs will load seamlessly with a different number is a frequent pitfall. Checkpoint formats are often tied to the cluster topology and may require conversion to be used with a different configuration.
- Neglecting Shared Resource Impact: On a multi-tenant cluster, writing large checkpoints can create I/O contention on a shared filesystem, slowing down your job and others. It's important to measure and account for this overhead.
What to learn next
- Distributed Training LLM: Data, Tensor, and Pipeline Parallelism Explained: Dive deeper into the parallelism strategies that make checkpoint sharding a necessity.
- GPU Cost Optimization: A Practical Guide to Reduce Your Bill: Learn how an effective checkpointing strategy fits into a broader framework for managing the cost of large-scale training.
- Mixed Precision Training Explained: How It Works: Understand why saving the gradient scaler state is a critical part of a complete checkpoint.
Sources and further reading
- Checkpointing in NeMo AutoModel
- Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures
- Convergence-aware optimal checkpointing for exploratory deep learning
- Checkpointing in NVIDIA NeMo Megatron
- An Asynchronous and Coordinated Local-First Checkpointing Scheme for Large-Scale Deep Learning
