Large language models (LLMs) have demonstrated remarkable capabilities, but their size makes them expensive to run and slow to respond. LLM knowledge distillation is a powerful technique for creating smaller, more efficient models that capture the abilities of these larger systems. It works by training a compact "student" model to imitate a powerful "teacher" model, making high-performance AI accessible for a wider range of applications.
Key takeaways
- Knowledge distillation trains a small "student" LLM to reproduce the behavior of a large "teacher" LLM, significantly reducing inference costs.
- Distillation is not just about copying answers; students can learn from the teacher's output probabilities (logits), generated text, or even internal patterns like attention maps (TinyBERT: Distilling BERT for Natural Language Understanding).
- A student model can match or exceed its teacher on a specific task, but this does not mean it has inherited the teacher's full reasoning, factuality, or safety capabilities (Survey on Knowledge Distillation for Large Language Models).
- Success depends on the distillation method, data quality, and a rigorous evaluation that tests for task performance and safety, not just how well the student mimics the teacher.
- Distillation involves a trade-off: it reduces inference costs but requires a significant upfront computational investment to run the teacher model and train the student (Towards safe and trustworthy knowledge distillation: a survey of existing methods and future directions).
What Is Knowledge Distillation for LLMs?
LLM knowledge distillation is a training method where a smaller, efficient "student" model learns to replicate the outputs of a larger, more capable "teacher" model. The goal is to transfer the teacher's specialized knowledge into a model that is cheaper, faster, and requires less memory. This makes it possible to deploy powerful AI on devices with limited resources or in applications that demand low latency.
The key idea behind distillation is that a teacher model's full output provides more information than just its final answer. For example, when predicting the next word, a powerful model doesn't just know the single best word; it assigns a probability to every possible word in its vocabulary. This distribution reveals which alternatives the model considered plausible. This nuanced information is a rich training signal that helps the student learn the teacher's patterns more effectively than training on simple text alone (Survey on Knowledge Distillation for Large Language Models).
By using a powerful but expensive teacher to generate high-quality training data and supervision signals, we can train capable small language models without having to build and train them from scratch.
How Does LLM Knowledge Distillation Work? (A Step-by-Step Guide)
The LLM knowledge distillation process involves using a teacher model to generate supervision signals that train a student model. The process can be broken down into seven main steps, from selecting the models to evaluating the final student.
Step 1: Choose Teacher and Student Models
The first step is to select your models. The teacher is typically a large, state-of-the-art model known for its high performance. The student is a much smaller model with a more efficient architecture, chosen to meet specific deployment constraints like latency or memory usage. The architectural differences, such as the number of layers, will influence how knowledge is transferred (On the Transformer Growth for Progressive BERT Training).
Step 2: Prepare the Distillation Dataset
You need a dataset of prompts to feed the teacher model. This can be an existing dataset for a specific task (like summarization or question-answering) or a large set of general prompts. Often, practitioners use a powerful model to synthetically generate a diverse set of high-quality prompts and completions to create a custom dataset tailored for distillation.
Step 3: Collect Teacher Signals
Next, you run the teacher model on every prompt in your distillation dataset to collect the training signals for the student. This is the most computationally intensive part of the process. The signal you collect depends on your chosen distillation method, but common signals include:
- Generated Text: The final text output from the teacher.
- Logits: The raw, unnormalized scores the teacher assigns to each possible next token. This is a very rich signal.
- Internal States: Intermediate representations like hidden states or attention patterns from within the teacher model (TinyBERT: Distilling BERT for Natural Language Understanding).
Step 4: Define the Distillation Objective (Loss Function)
A loss function is a formula that measures the difference between the student's output and the teacher's signal. The training process aims to minimize this difference. For example, if you are using logits, a common loss function is the Kullback-Leibler (KL) divergence, which measures how one probability distribution differs from a second, reference distribution. This distillation loss is often combined with a standard supervised loss (like cross-entropy on the ground-truth answer) to ground the student's learning.
Step 5: Train the Student Model
With the teacher's signals and a loss function, you can now train the student. During training, the student model processes the same prompts as the teacher. Its outputs are compared to the teacher's stored signals using the loss function. The model's parameters are then adjusted to minimize the loss, gradually teaching the student to mimic the teacher's behavior. This is often done using the same techniques as standard model training, such as backpropagation and gradient descent.
Step 6: Evaluate the Student
After training, the student must be evaluated rigorously. Crucially, evaluation should not just measure how well the student agrees with the teacher. High agreement doesn't always lead to the best performance on real-world tasks (Survey on Knowledge Distillation for Large Language Models). A thorough evaluation includes:
- Task Performance: How well the model performs on an independent test set for the target task.
- Safety and Factuality: Checking for biases, harmful content, and hallucinations, which can be inherited from the teacher (Towards safe and trustworthy knowledge distillation: a survey of existing methods and future directions).
- Efficiency Metrics: Measuring inference speed, memory usage, and cost to ensure it meets deployment goals.
Step 7: Decide on Further Task Adaptation
The final step is to decide the student model's path forward. A general-purpose distilled model can be further fine-tuned for various downstream tasks. Alternatively, the entire distillation process can be performed specifically for one task, which may yield higher performance on that task at the cost of the model's general capabilities (Survey on Knowledge Distillation for Large Language Models).
What Are the Main Distillation Methods?
There are two primary categories of LLM knowledge distillation methods, defined by the type of signal the student learns from: response-based and feature-based. The best method depends on your level of access to the teacher model and the specific capabilities you want to transfer.
Response-Based Distillation (Black-Box)
In this approach, the student learns from the teacher's final outputs. This is ideal when you only have API access to the teacher model and cannot inspect its internal workings.
- Sequence Distillation: The simplest method. Here, the teacher generates text completions for a set of prompts. These completions are then used as the target outputs for training the student. The student learns to generate text that is stylistically and factually similar to the teacher's.
- Logit Distillation: This method uses the teacher's full probability distribution over the next token (the logits) as the training signal. This is a richer signal than just the final text because it tells the student not only what the teacher's top choice was but also what it considered to be close alternatives. This requires access to the teacher's logits.
Feature-Based Distillation (White-Box)
This approach requires full access to the teacher model's architecture and weights. The student learns to mimic the teacher's internal computations, such as its hidden states or attention patterns. These methods can be very effective but are more complex to implement, especially if the teacher and student have different architectures. For instance, methods like TinyBERT distill knowledge from multiple layers of the teacher, including embeddings and attention maps (TinyBERT: Distilling BERT for Natural Language Understanding), while MiniLM focuses on distilling the self-attention mechanism itself (MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers).
| Method | Signal Used | Teacher Access Required | Signal Richness | Common Use Case |
|---|---|---|---|---|
| Sequence Distillation | Generated text outputs | API (Black-box) | Low | Training a student on teacher-generated examples when only text output is available. |
| Logit Distillation | Output probability distributions (logits) | Logit access (Grey-box) | High | Transferring nuanced word-choice probabilities from a teacher. |
| Feature Distillation | Internal hidden states, attention maps | Full model (White-box) | Very High | Aligning student's internal representations with a teacher of a similar architecture. |
A Worked Example: Distilling with Teacher Logits
To make the concept of logit distillation concrete, let's walk through a simplified example. Imagine we want to train a student to predict the next word after the phrase "The cat sat on the __".
Scenario:
- Vocabulary: For simplicity, let's say our vocabulary is just four words:
{'mat', 'rug', 'hat', 'sky'}. - Ground Truth: The correct word is
mat.
1. Get Teacher Logits and Probabilities
We feed the prompt to our large teacher model. It produces a logit for each word in the vocabulary. Logits are the raw, unnormalized scores a model produces.
- Teacher Logits:
{'mat': 4.0, 'rug': 3.8, 'hat': 1.0, 'sky': -2.0}
To turn these into probabilities, we use the softmax function. At a temperature (T) of 1, the probabilities reflect the model's confidence. Temperature is a parameter that scales logits to control randomness.
- Teacher Probabilities (T=1):
{'mat': 0.54, 'rug': 0.44, 'hat': 0.02, 'sky': 0.00}The teacher is quite confident in "mat" but also sees "rug" as a very plausible alternative.
2. Soften Probabilities with Temperature
For distillation, we use a higher temperature (e.g., T=2) to "soften" the probabilities. This amplifies the signal from less likely but still plausible options, like "hat," providing more information to the student.
- Teacher "Soft" Probabilities (T=2):
{'mat': 0.39, 'rug': 0.36, 'hat': 0.16, 'sky': 0.09}Notice that "hat" now has a much higher probability, teaching the student that it's a more reasonable guess than "sky," even if it's not the best one.
3. Get Student Logits
Now, we give the same prompt to our untrained student model. Its initial logits are less confident and more uniform.
- Student Logits:
{'mat': 2.5, 'rug': 2.4, 'hat': 2.0, 'sky': 1.9} - Student "Soft" Probabilities (T=2):
{'mat': 0.27, 'rug': 0.26, 'hat': 0.24, 'sky': 0.23}
4. Calculate the Loss
The training process has two goals, which are combined into a single loss function:
- Distillation Loss: Make the student's soft probabilities match the teacher's soft probabilities. This is often calculated using KL Divergence, which measures the difference between the two distributions.
- Supervised Loss: Make the student correctly predict the ground-truth label ("mat"). This is calculated using Cross-Entropy Loss against the "hard" label (where "mat" is 1 and all others are 0).
The final loss is a weighted sum:
Total Loss = α * Distillation_Loss + (1-α) * Supervised_Loss
The model's parameters are adjusted to minimize this total loss. Over many steps, the student learns to produce probability distributions similar to the teacher's while also staying grounded in the correct answers.
Can a Small Student Model Truly Match a Large Teacher?
Yes, a small student model can match or even surpass its large teacher's performance, but this success is typically limited to a specific task or benchmark (Survey on Knowledge Distillation for Large Language Models). If you distill a large model's knowledge for a single function, like summarizing medical reports, the resulting student can become an expert at that narrow task. However, this does not mean the student has inherited the teacher's full range of capabilities.
A distilled model is not a miniature clone of its teacher. It is a new model optimized to replicate specific behaviors. It is not established that distillation can transfer the teacher's broad world knowledge, complex reasoning abilities, or robustness to unexpected inputs. The student learns a shortcut to produce the right answers for the distillation task, which might not generalize well to different domains or prompt styles.
Furthermore, a student's performance is highly dependent on:
- Student Capacity: The student model must be large enough to represent the knowledge being transferred.
- Data Quality: The diversity and quality of the distillation dataset are critical for generalization.
- Distillation Method: The chosen signals (logits, text, features) determine what kind of knowledge is transferred.
Ultimately, while a student may achieve high fidelity to its teacher on the training data, this does not guarantee better real-world performance. Independent and comprehensive evaluation is essential to understand a distilled model's true capabilities and limitations (Survey on Knowledge Distillation for Large Language Models).
When Not to Use Knowledge Distillation
While powerful, LLM knowledge distillation is not always the right solution. Here are some scenarios where it might be inappropriate or counterproductive:
- When you need general, open-ended capabilities: If your application requires a model that can handle a vast and unpredictable range of topics and reasoning tasks, a distilled model focused on a narrow domain will likely fail.
- When the teacher model has known flaws: Distillation transfers the teacher's behavior, including its biases, factual errors, and safety vulnerabilities. If the teacher is known to be unreliable in certain areas, the student will likely inherit those same problems.
- When you cannot afford the upfront computation: The distillation process itself is expensive. It requires running millions of inferences with a large teacher model and then fully training a student model, which can be a significant computational investment.
- When a simpler method is sufficient: For some tasks, simply fine-tuning a pre-existing small model on a high-quality dataset may achieve "good enough" performance with far less complexity and cost than setting up a full distillation pipeline.
What to learn next
- LoRA Fine-Tuning: A Complete Guide to Low-Rank Adaptation: Explore another popular and efficient method for adapting large models for specific tasks.
- Fine Tuning vs RAG: Which Method Should You Choose?: Compare two different approaches for customizing LLM knowledge and behavior.
- LLM Quantization Explained: A Practical Guide: Learn about another key technique for reducing model size and improving inference speed.
