The LLM context window is the amount of information a model can process at one time. This limit, measured in tokens, dictates how much text—from user queries to background documents—can be included in a single request. Understanding its size and performance characteristics is a factor in developing AI applications.

Key takeaways

  • An LLM context window is the total number of tokens a model can process at once, including all input and often the generated output.
  • A model's advertised context capacity is not the same as its effective performance; models often struggle with information placed in the middle of long prompts.
  • Larger context windows are not always better, as they can introduce distracting or conflicting information that harms accuracy.
  • Engineers must budget token usage, allocating space for system prompts, chat history, retrieved documents, and the user's query within the model's limit.
  • Exceeding the context limit does not cause the model to "forget." It typically results in an API error or silent truncation of older information by the application.

What Is an LLM Context Window?

An LLM context window is the maximum number of tokens a model can handle in a single input and output cycle. This window acts like a temporary workspace. For each new request, you fill this workspace with all the necessary information: instructions, previous messages, reference documents, and the latest user query. The model can only see what is currently inside this window.

A token is the basic unit of text for a model, often a word, part of a word, or punctuation. For example, "LLM context window" might be tokenized into ["LLM", "Ġcontext", "Ġwindow"]. The limit is measured in these tokens, not words or pages (Source).

Crucially, the context window is not persistent memory. Once a request is complete, the workspace is cleared. The model does not remember past conversations unless you explicitly include that history in the context for the next turn. This process is part of the underlying Transformer architecture, which processes all tokens in the window simultaneously.

How a Context Window Works, Step by Step

A model processes the context window through a series of steps to understand relationships between different pieces of information and generate a coherent response. This happens for every single request sent to the model.

  1. Tokenization: First, all inputs are converted into a sequence of numerical tokens. This includes the system prompt (e.g., "You are a helpful assistant"), the conversation history, any provided documents, and the user's latest question.
  2. Combination: These separate token sequences are combined into a single, long list. This flat sequence is the final input that will be sent to the model.
  3. Positional Encoding: Because the core model architecture does not inherently understand word order, it adds positional information to each token's representation. This tells the model where each token is located in the sequence (e.g., this is the 5th token, this is the 1024th token).
  4. Attention Processing: The model processes the entire sequence at once. The self-attention mechanism allows each token to look at and weigh the importance of every other token in the context window. This is how the model builds a contextual understanding, connecting a pronoun in the user's query to a name mentioned much earlier in the text.
  5. Output Generation: The model predicts the next most likely token. This new token is appended to the input sequence, and the process repeats. The model uses this updated sequence to predict the next token, and so on, until it generates an end-of-sequence token or hits a predefined output limit.

Anatomy of a Prompt: A Worked Example of Token Budgeting

To make this concrete, imagine a customer support chatbot using a 4,096-token context window. The chatbot needs to answer a user's question using a retrieved knowledge base article. Every part of the prompt consumes a portion of the total token budget.

Here is an illustrative breakdown of the token costs:

  • System Prompt (Tells the model its role): 50 tokens
    • "You are a friendly support agent for ExampleCorp. Use the provided document to answer the user's question. Be concise."
  • Chat History (Provides short-term memory): 300 tokens
    • Let's assume the last three turns of conversation are included, averaging 100 tokens each.
  • Retrieved Document (Provides the necessary facts): 1,500 tokens
    • A relevant snippet from the company's knowledge base about product returns.
  • User Question (The immediate query to answer): 30 tokens
    • "Okay, thanks. How long do I have to return my purchase if I bought it online?"

Now, let's calculate the total input budget used:

50 (System) + 300 (History) + 1,500 (Document) + 30 (Question) = 1,880 input tokens

The model's response will also consume tokens from the same window. The remaining budget for the output is:

4,096 (Total) - 1,880 (Input) = 2,116 tokens for the answer

This is plenty of room for a detailed response. However, if the model only had a 2,048-token context window (Source), the 1,880-token input would leave only 168 tokens for the answer. To avoid an error, the application would need to shorten the chat history or the document snippet.

Why Does Accuracy Drop in the Middle of Long Prompts?

Even if a prompt fits within the LLM context window, models may not use all the information equally well. Research has identified a "lost-in-the-middle" effect, where models recall information from the beginning and end of a long context more reliably than from the middle (Source). This results in a U-shaped performance curve.

This phenomenon was demonstrated using "needle-in-a-haystack" tests. In these evaluations, a key piece of information (the "needle") is placed at various positions within a long, distracting document (the "haystack"). When the model is asked to retrieve this information, its accuracy is highest when the needle is at the very start or very end of the document. Performance drops significantly when the needle is buried in the middle.

Proposed causes include biases in the attention mechanism and the way positional information is encoded, which may favor the ends of the sequence (Source).

For engineers, this has a clear practical takeaway: place your most important instructions and data at the start or end of the prompt. For example:

  • Start of Prompt: Put critical instructions, rules, or persona definitions in the system prompt.
  • End of Prompt: Place the user's most recent question and any highly relevant, just-retrieved data immediately before it.

Is a Larger Context Window Always Better?

Increasing the context window from a few thousand to over 100,000 tokens may seem like a clear advantage, but it comes with significant trade-offs (Source). A larger context window is not always the best solution and can sometimes harm performance.

A primary downside is the introduction of distracting or irrelevant information. A longer prompt can include contradictory material that confuses the model, leading to less accurate or unfocused answers (Source). For some tasks, providing a smaller amount of highly relevant information is more effective than providing a larger, less-filtered context.

An alternative strategy to using a very long context is Retrieval-Augmented Generation (RAG).

Comparison: Long Context vs. RAG

Feature Long-Context Prompting Retrieval-Augmented Generation (RAG)
How it Works The entire document or a very long history is fed directly into the context window. An external system first finds and retrieves only the most relevant document snippets.
Pros Can be used to synthesize information from many different parts of a large document. Reduces distraction by selecting and providing only the most relevant text.
Cons Can suffer from the "lost-in-the-middle" effect. Depends on the quality of the retrieval step; may miss context from un-retrieved parts.
Best for... Analyzing entire documents (e.g., legal review, book summary) where broad context is key. Question-answering systems, where specific facts need to be pulled from a large knowledge base.

Choosing between these strategies depends on the specific task. You can learn more about how models produce answers from text in our guide on How Are Large Language Models Trained? A Complete Guide.

What Happens When You Exceed the Context Limit?

When the combined length of your input and desired output exceeds the LLM context window, the model does not simply "forget." The outcome is determined by the model's API and the application logic built around it.

Common behaviors include:

  1. API Error: The most straightforward result. The API service rejects the request and returns an error message stating that the input exceeds the maximum context length. This forces the developer to handle the issue by shortening the prompt.
  2. Truncation: This is a common strategy for managing conversational context. The application automatically removes the oldest messages from the chat history to make room for the new ones. This is a "first-in, first-out" approach that keeps the conversation flowing but loses long-term memory.
  3. Sliding Window: A more formal version of truncation where the application is always given the most recent N tokens. As the conversation continues, the window "slides" forward, ensuring the model always has the most immediate context.
  4. Summarization: Advanced applications might try to summarize older parts of the conversation and include the summary in the prompt instead of the full transcript. This can preserve key information but risks losing important details or introducing inaccuracies.

The key takeaway is that context management is often an application-level responsibility. The model itself only processes what it is given in a single request.

Common Mistakes When Using Context Windows

Avoiding common pitfalls can significantly improve the reliability and efficiency of your application.

  • Mistake: Treating the advertised limit as a guarantee of performance.
    • Fix: Test your use case with important information placed at the beginning, middle, and end of the prompt to check for "lost-in-the-middle" issues.
  • Mistake: Using a single "words per token" rule of thumb.
    • Fix: Use an official tokenizer library to get the exact token count for your text. Tokenization varies significantly between languages, code, and even punctuation.
  • Mistake: Filling the context with irrelevant "noise."
    • Fix: If using a retrieval system, refine your retrieval logic to pull in only the most relevant information. More data is not always better.
  • Mistake: Forgetting to budget for the output.
    • Fix: Always subtract the input token count from the total context window size to know how much space is left for the model's response, as the budget often covers both input and generation.

What to learn next

Sources and further reading