When you interact with a Large Language Model (LLM), you might notice that asking the same question twice can yield different answers. This variability isn't random chaos; it's a controlled process governed by sampling parameters. Understanding LLM temperature and top p is key to managing the trade-off between the model's creativity and its consistency.
This guide explains how these parameters work, from the underlying math to a practical tuning workflow, so you can get the output you need.
Key takeaways
- Temperature controls randomness by reshaping probabilities. A low temperature (e.g., 0.2) sharpens the probability distribution, making the model’s output more deterministic by strongly favoring the most likely next words. A high temperature (e.g., 1.5) flattens the distribution, increasing word diversity but also the risk of errors (OpenAI API Reference).
- Top-p (Nucleus Sampling) controls diversity by limiting the candidate pool. This method selects the smallest possible set of words whose combined probability meets a certain threshold
p. This creates an adaptive list of candidates for the next word, allowing for diversity when the model is uncertain but precision when it is confident (OpenAI API Reference). - These are controls for randomness, not factuality. Setting a low temperature or top-p does not make an LLM more truthful. It only makes the model's most likely response more repeatable, regardless of whether that response is correct (OpenAI Help Center). You cannot rely on sampling to prevent model hallucinations.
- Start by tuning only one parameter. Temperature and top-p can interact in complex ways. To get predictable results, it's best to modify either temperature or top-p, but not both at the same time. One major API provider explicitly recommends this approach for easier tuning (OpenAI API Reference).
- Temperature 0 does not guarantee identical output. While setting temperature to 0 aims for the least-random decoding, it is not an absolute guarantee of reproducibility. Factors like provider-side updates, non-associative floating-point math, and how the model breaks ties between equally probable tokens can still cause slight variations (OpenAI Help Center).
How LLM Sampling Works: From Logits to Tokens
To understand LLM temperature and top-p, you first need to know how a model chooses its next word. This process, known as sampling or decoding, happens for every single token the model generates and follows four main steps.
A token is a piece of text, like a word or part of a word, that the model processes. You can learn more in our guide to LLM tokenization.
The model generates logits. After processing the input, the model produces a raw score, called a logit, for every possible next token in its vocabulary. A logit is an un-normalized number that represents the model's relative preference for that token being the next one in the sequence. Higher logits mean higher preference.
Logits are converted to probabilities. The entire list of logits is passed through a mathematical function called softmax. Softmax converts the raw scores into a probability distribution, where each token is assigned a probability between 0 and 1, and the probabilities of all tokens in the vocabulary add up to 1.
Sampling parameters are applied. This is where temperature and top-p come into play. Temperature adjusts the logits before the softmax step, making the final distribution sharper or flatter. Top-p filters the token list after the softmax step, creating a smaller candidate pool.
A token is selected. The model samples a token from the final, adjusted pool of candidates according to their assigned probabilities. This token is added to the output, and the entire process repeats to generate the next token.
This token-by-token generation is why small changes in sampling can lead to vastly different final outputs, as the changes compound at each step.
What Is Temperature and How Does It Reshape Probabilities?
Temperature directly controls the randomness of the output by altering the shape of the token probability distribution. It is a divisor applied to the logits before the softmax function. A lower temperature makes the model more confident in its top choices, while a higher temperature encourages it to explore more options, even less likely ones.
Here is the mathematical intuition: softmax(logits / temperature).
- Low Temperature (e.g., 0.2 to 0.7): Dividing the logits by a number less than 1 makes the large logits even larger relative to the small ones. This sharpens the probability distribution, causing the model to strongly favor the few tokens with the highest scores. This is useful for tasks that require precision and predictability, like code generation or factual summarization (OpenAI Help Center).
- Effect: More focused, conservative, and repeatable output.
- Temperature of 1.0: Dividing by 1 leaves the logits unchanged. The model samples from the "natural" probability distribution calculated by the softmax function. This is a neutral, balanced setting.
- High Temperature (e.g., > 1.0): Dividing the logits by a number greater than 1 squashes them closer together. This flattens the probability distribution, making less likely tokens more competitive with the top choices. This is useful for creative tasks like writing poetry or brainstorming ideas, but it also increases the chance of the model producing nonsensical text or factual errors (OpenAI API Reference).
- Effect: More diverse, creative, and random output.
Setting temperature to 0 is a special case that typically makes the model select the token with the highest logit every time (greedy decoding). However, perfect reproducibility isn't guaranteed.
What Is Top-P (Nucleus Sampling) and How Does It Work?
Top-p sampling, also known as Nucleus Sampling, controls output diversity by creating a dynamic and adaptive list of candidate tokens. Instead of reshaping probabilities like temperature, top-p works by truncating the probability distribution, ensuring that the sampling pool is neither too large (leading to nonsense) nor too small (leading to repetitive text).
Here’s how it operates:
- After calculating the probabilities for all tokens (potentially adjusted by temperature), the model sorts them from most likely to least likely.
- It then goes down the list, summing their probabilities until the cumulative total reaches the threshold
p. - The tokens included in this sum form the "nucleus." All other tokens are discarded for the current generation step.
- The model then samples a token only from this nucleus.
The key advantage of top-p is its adaptive nature. If the model is very confident about the next token, the probability distribution will be sharply peaked. For example, the token "hot" might have a 95% probability. With a top_p of 0.9, the nucleus will contain only that single token. But if the model is uncertain, the probabilities will be spread out. The nucleus might then include a dozen or more tokens, allowing for more diversity exactly when it's needed (OpenAI Help Center).
- Low Top-P (e.g., 0.1): Very restrictive. Only the most probable tokens are considered. Leads to safe but potentially repetitive text.
- High Top-P (e.g., 0.9): Very permissive. A wide range of tokens is considered, increasing diversity. A value of 1.0 effectively disables this method, meaning no tokens are discarded from the distribution based on their cumulative probability.
Top-p was developed as an alternative to an older method, top-k, which always samples from a fixed number (k) of top tokens (Survey of Decoding Strategies). The candidate list size for top-p adapts to the context, unlike the fixed-size list of top-k.
A Worked Example: Visualizing Temperature and Top-P
Let's make this concrete. Imagine after the prompt "The cat sat on the...", the model generates logits for a small vocabulary.
Illustrative Logits:
mat: 3.2floor: 2.8couch: 2.1sky: -1.5
Step 1: Default Probabilities (Temperature = 1.0)
With a temperature of 1.0, we apply the softmax function directly to the logits.
| Token | Logit (T=1.0) | Probability |
|---|---|---|
mat |
3.2 | ~49.7% |
floor |
2.8 | ~33.3% |
couch |
2.1 | ~16.5% |
sky |
-1.5 | ~0.4% |
Here, mat is the favorite, but floor is also a strong contender.
Step 2: Low Temperature (T = 0.5)
Now, let's set the temperature to 0.5. We first divide each logit by 0.5 before applying softmax. This sharpens the distribution.
| Token | Logits / 0.5 | Probability |
|---|---|---|
mat |
6.4 | ~64.1% |
floor |
5.6 | ~28.8% |
couch |
4.2 | ~7.1% |
sky |
-3.0 | ~0.0% |
The probability for mat jumped significantly, making it much more likely to be picked. The model becomes more deterministic.
Step 3: High Temperature (T = 1.5)
If we set the temperature to 1.5, we divide the logits by 1.5, flattening the distribution.
| Token | Logits / 1.5 | Probability |
|---|---|---|
mat |
2.13 | ~43.5% |
floor |
1.87 | ~33.6% |
couch |
1.40 | ~21.0% |
sky |
-1.00 | ~1.9% |
Now, mat is still the most likely, but floor and couch are much more competitive. Even the unlikely token sky has a small chance. The model is more likely to generate a diverse response.
Step 4: Applying Top-P (p = 0.90)
Let's go back to our T=1.0 probabilities and apply top_p=0.90.
- Sorted probabilities:
mat(49.7%),floor(33.3%),couch(16.5%),sky(0.4%). - Cumulative sum:
mat: 49.7% (which is < 90%)mat+floor: 49.7% + 33.3% = 83.0% (which is < 90%)mat+floor+couch: 83.0% + 16.5% = 99.5% (which is > 90%)
- The nucleus is {
mat,floor,couch}. - The model will only sample between
mat,floor, andcouch, completely ignoringsky.
Temperature vs. Top-P vs. Top-K: A Comparison
These three decoding strategies all control randomness, but they do so in different ways.
| Parameter | How it Works | Key Characteristic |
|---|---|---|
| Temperature | Rescales the logits before softmax, altering the probability of every token in the vocabulary. | Modifies the entire distribution's "sharpness" or "flatness" (Survey of Decoding Strategies). |
| Top-P | Selects a variable number of tokens based on their cumulative probability mass. | Adaptive: The size of the candidate pool changes based on the model's confidence. |
| Top-K | Selects a fixed k number of the most likely tokens. |
Fixed: The size of the candidate pool is always k, regardless of confidence. |
Temperature and top-p can be used in combination. The adaptive nature of top-p was intended to avoid some of the limitations of using a fixed-size candidate set like top-k.
A Safe Tuning Workflow for Beginners
Tuning sampling parameters can feel like a dark art, but a methodical approach yields the best results.
Define Your Goal First: What are you trying to achieve?
- Consistency: For tasks like data extraction, summarization, or code generation, you want predictable output. Aim for low temperature (e.g., 0.0 to 0.4).
- Creativity: For tasks like brainstorming, content creation, or building a conversational chatbot, you want varied and interesting responses. Aim for higher temperature (e.g., 0.7 to 1.2) or a high top-p (e.g., 0.9).
Start with Defaults and Change One Thing: Begin with the provider's default settings. Then, choose either temperature or top-p to adjust. The general recommendation is to tune temperature first. Trying to tune both simultaneously makes it difficult to understand which parameter is causing the change (OpenAI API Reference).
Use a Fixed Prompt and Multiple Runs: Test your changes using the exact same input prompt. Run it 3-5 times to observe the range of variation in the output. A single result can be misleading.
Evaluate and Iterate: Did the change move you closer to your goal? If you need more consistency, lower the temperature. If you need more variety, raise it. Make small, incremental adjustments and re-test.
Consider Top-P for Advanced Control: Once you have a good feel for temperature, you can experiment with top-p. A common strategy is to use a moderate temperature (e.g., 0.7) and then constrain the output with a top-p of 0.9 to cut off the long tail of low-probability tokens.
What to learn next
Understanding sampling is a great step toward mastering LLMs. To continue learning about the internal workings of these models, consider these topics:
- Transformer Architecture Explained: Learn about the core architecture that powers modern LLMs, from self-attention to token processing.
- Why Do LLMs Hallucinate?: Explore the reasons behind model-generated falsehoods and how they relate to the training process, not just sampling.
- An Explanation of the LLM Context Window: Discover how the amount of text a model can "see" at once impacts its performance and coherence.
