Large language models (LLMs) are incredibly expensive to train, requiring massive amounts of data and computational power. Before committing to a training run, researchers need to predict if the investment will yield the expected improvements. This is where LLM scaling laws come in. They provide a framework for estimating a model's future performance based on the resources allocated to its training.
These 'laws' are empirical observations, not laws of nature, which researchers use to guide resource allocation and estimate training needs.
Key takeaways
- LLM scaling laws are formulas that show how a model's performance improves as you increase its size (parameters), the amount of training data (tokens), or the computational budget (FLOPs).
- These relationships often follow a power law, meaning performance improves predictably as resources are scaled up, with the relationship appearing linear on a log-log plot.
- The "Chinchilla" paper showed that for a fixed compute budget, a smaller model trained on more data often outperforms a larger model trained on less data (Source). This corrected a previous view that favored rapidly increasing model size.
- Compute-optimal training is the process of finding the ideal balance between model size and dataset size to achieve the lowest possible loss for a given compute budget.
- Scaling laws are approximations, not universal truths. Their predictions can break down if you change the model architecture, data quality, or training methods (Source).
What Are LLM Scaling Laws?
LLM scaling laws are empirical formulas that predict a model's performance based on the scale of three key resources: model size, dataset size, and compute. Performance is typically measured by the model's loss—a score that indicates how well it predicts the next token in a sequence. A lower loss means better performance. These laws help researchers answer a critical question: how should we allocate a fixed budget to get the best possible model?
Think of it like baking a cake. The quality of your cake (loss) depends on the size of your oven (model parameters), the amount of ingredients (training data), and how long you bake it (compute). Scaling laws are like a baker's chart that helps you find the best recipe for the resources you have. Adding more ingredients without a big enough oven might result in an under-baked cake.
The three core variables are:
- Model Size (N): The number of parameters in the model. Parameters are the internal variables, or weights, that the model learns during training.
- Dataset Size (D): The number of tokens in the training dataset. A token is a piece of text, roughly equivalent to a word or sub-word.
- Compute (C): The total amount of computation used for training, measured in FLOPs (Floating-point Operations).
Researchers found that the relationship between these variables and the model's final loss often follows a power law (Source). This means that as you increase N, D, or C, the loss L decreases according to a formula like:
L(N) ∝ 1 / N^α
Here, α (alpha) is a small positive number called the scaling exponent. When you plot these relationships on a log-log graph, they form a straight line, which makes performance at larger scales predictable within a certain range.
How Do Researchers Discover Scaling Laws?
Researchers discover scaling laws through a systematic, multi-step experimental process. The goal is to isolate the effect of scaling one variable (like model size) while keeping everything else as constant as possible. This allows them to observe clear trends and fit a mathematical formula to the results.
Here is the typical process:
Define the Metric: The first step is to choose a consistent metric for "performance." Most scaling law research uses cross-entropy loss on a held-out test set. This metric measures, on average, how surprised the model is by the next token in a sequence. Lower surprise means lower loss and a better model.
Run Many Controlled Experiments: Researchers train dozens of models at varying scales. For example, they might train models with 10 million, 100 million, 1 billion, and 10 billion parameters. For each model size, they might also vary the amount of training data, perhaps using 1 billion, 10 billion, and 100 billion tokens. It is crucial that other factors, like the model's architecture and the training process, remain the same across experiments (Source).
Plot the Results: After all models are trained, researchers plot the final loss against the variable they scaled (e.g., parameter count) on a log-log plot. On a log-log plot, a power-law relationship appears as a straight line. The straightness of this line indicates how well the scaling "law" holds.
Fit a Power Law: Using statistical regression, a line is fitted to the data points on the log-log plot. The slope of this line gives researchers the scaling exponent (the
αin the formula). This equation can then be used to predict the loss for models that are even larger than those trained in the experiment.Validate by Extrapolation: To test the predictive power of the formula, researchers train a final, much larger model that was not used to create the formula. If this model's performance lands on or near the predicted line, it validates the scaling law for that range.
This process allows teams to make informed decisions and budget for massive training runs with a reasonable expectation of the outcome.
What Does Compute-Optimal Training Mean?
Compute-optimal training means finding the best combination of model size (parameters) and dataset size (tokens) to achieve the lowest possible loss for a fixed computational budget. It treats training as an optimization problem where the goal is to get the most performance for your investment, rather than just building the biggest model possible (Source).
The total compute C required for training can be roughly estimated by the formula C ≈ 6 * N * D, where N is the number of parameters and D is the number of training tokens. This formula shows that you can spend your compute budget C in different ways: a huge model (N) trained on little data (D), a small model trained on huge data, or something in between.
Two landmark papers proposed different answers to this allocation problem.
Kaplan et al. (2020)
An early paper from OpenAI by Kaplan et al. suggested that model size was the most important factor. Their analysis concluded that as the compute budget grows, the model's parameter count should grow much more quickly than the number of training tokens (Source). This led to a trend of training very large models on relatively small datasets.
Hoffmann et al. (2022, "Chinchilla")
A later paper from DeepMind by Hoffmann et al. re-evaluated this relationship. Their study, nicknamed "Chinchilla," found that many large models were significantly "undertrained." They concluded that for optimal performance, model size and training tokens should scale at roughly the same rate (Source). This implies that as the total compute budget increases, both model size and the number of training tokens should be increased at roughly comparable rates to maintain optimality.
The table below summarizes the different recommendations.
| Aspect | Kaplan et al. (2020) | Hoffmann et al. (2022, Chinchilla) |
|---|---|---|
| Primary Finding | Model size (N) is more important than dataset size (D). |
N and D should be scaled in proportion. |
| Compute Allocation | Prioritizes increasing N much faster than D. |
Recommends N and D grow at similar rates. |
| Scaling Exponents | N ∝ C^0.73 and D ∝ C^0.27 (Source) |
N ∝ C^0.50 and D ∝ C^0.50 (Source) |
| Implication | Build the largest model possible for your compute budget. | Build a smaller model but train it on much more data. |
The Chinchilla findings have been highly influential, shifting the industry's focus toward curating larger, higher-quality datasets and training more "compute-optimal" models.
Worked Example: Allocating a Compute Budget
To make compute-optimal training concrete, let's consider a conceptual example. Imagine you have a fixed compute budget C. Using the approximation C ≈ 6 * N * D, how would you allocate this budget between parameters (N) and tokens (D) based on the two different scaling philosophies?
Scenario 1: Kaplan-style (Size-First)
This approach prioritizes model size over training data. The strategy would be to allocate the compute budget to maximize the parameter count, N. Given the relationship C ≈ 6 * N * D, spending more of the budget on a larger N necessarily means the dataset size D must be smaller. The result is a very large model trained on a comparatively modest amount of data.
Scenario 2: Chinchilla-style (Balanced)
This approach recommends scaling model size and dataset size at similar rates. To follow this strategy for the same fixed budget C, one would choose a smaller parameter count N than in the Kaplan-style scenario. This frees up the budget to train on a substantially larger number of tokens D. The result is a smaller model trained on a much larger dataset.
The Chinchilla study demonstrated that its approach produced a model with better loss and downstream results than larger, undertrained comparators at a similar compute budget (Source).
Why Don't Smooth Loss Curves Guarantee Better AI?
A smooth, predictable drop in pretraining loss is a powerful signal, but it does not automatically mean the model will be better at every specific task. The connection between this simple statistical metric and the complex, useful behaviors we want from AI is not always direct. Relying solely on scaling laws for loss can be misleading.
Here’s why:
Loss is an Average: Pretraining loss measures the average next-token prediction accuracy over a massive, diverse dataset. It is a good proxy for general language understanding, but it doesn't measure any specific skill directly. A model can get better at predicting common web text without necessarily improving its ability to perform multi-step reasoning or write safe code.
Capabilities Can Emerge: Some advanced capabilities, like arithmetic or answering questions about code, don't seem to improve smoothly. Instead, they can appear to "emerge" suddenly once a model reaches a certain scale (Source). A smoothly decreasing loss curve may hide these sharp, unpredictable jumps in performance on specific benchmarks.
Benchmark Performance is Noisy: While loss might decrease predictably, performance on specific downstream tasks (like question answering or summarization benchmarks) can be much more variable. One model with a slightly higher loss might outperform another on a specific task due to differences in its training data mixture or architecture.
Post-training Matters Immensely: The final, most useful versions of models are often the result of extensive post-training, including instruction fine-tuning and reinforcement learning from human feedback (RLHF). These crucial steps shape the model's behavior and abilities in ways that pretraining loss cannot capture. A model that is "training-optimal" is not necessarily the best foundation for creating a helpful and harmless assistant.
In short, scaling laws are an indispensable tool for managing the enormous cost of pretraining. But they predict a statistical average, not the full spectrum of behaviors that make an AI model truly useful.
What to learn next
- How Are Large Language Models Trained? A Complete Guide: Explore the full lifecycle of creating an LLM, from data collection and pretraining to fine-tuning and deployment.
- Mixture of Experts LLM: How MoE Models Work: Learn about a modern architecture that complicates simple scaling laws by using a different number of parameters for each input.
- LLM Tokenization Explained: A Complete Guide: Understand what "tokens" are and why they are the fundamental unit of data for both training and scaling laws.
