Some language models seem to "think" longer before answering a complex question. This isn't just a feeling; it's a specific design choice. This article provides a reasoning models explained guide, showing how they work, how they differ from standard LLMs, and when the trade-off of higher cost for better accuracy is worthwhile.

Key takeaways

  • What they are: Reasoning models are language models trained to use extra computation when you ask a question. They generate internal steps, or "thinking tokens," before providing a final answer.
  • The key trade-off: This extra deliberation can improve accuracy on complex, multi-step tasks like math and coding. However, it comes at the cost of higher latency and greater expense.
  • How they learn: Many reasoning models are trained using reinforcement learning. This method rewards the model for achieving verifiable outcomes, like getting a math problem right (Source). This encourages emergent behaviors like self-correction.
  • A critical caveat: The "reasoning" you see is a learned output, not a direct transcript of the model's computation. It can still contain errors or justifications written after the fact.

What Are Reasoning Models and How Do They Differ from Standard LLMs?

Reasoning models are language models that spend additional compute at inference time to deliberate before giving an answer. Unlike standard models that try to generate the final response immediately, reasoning models produce and evaluate intermediate steps. This behavior results from specialized training methods and specific configurations that allocate a "budget" for this extra thinking. They are not a new architecture but a different approach to training and using LLMs.

A standard, direct-answer language model is like a student who raises their hand instantly with the first answer that comes to mind. It's fast but can be superficial. A reasoning model is like a student who first uses scratch paper to work through the problem before giving the final answer. This process takes more time and effort but often leads to a more accurate result on difficult problems.

This distinction is not about model size. A smaller model trained for reasoning can outperform a larger, general-purpose model on specific logical tasks. The key difference lies in the use of test-time compute: the resources spent during the request to explore possible solutions. This is different from the compute used during the initial training of the large language model. For example, OpenAI's o1 series models are explicitly trained to use a long internal chain of thought before answering (Source).

Reinforcement-Trained Models vs. Chain-of-Thought Prompting

It's important to distinguish between a model architecturally trained for reasoning and a standard model prompted to show its work.

  • Chain-of-thought prompting is a technique where a user explicitly asks any capable model to "think step by step" to guide its output.
  • A reinforcement-trained reasoning model, however, has been specifically optimized during its training to discover and use effective deliberation patterns internally, often without being prompted to.

The approaches can be combined, but the underlying capability is distinct (Source). This article focuses on models with built-in reasoning capabilities.

How Do Reasoning Models 'Think'? The Inference-Time Compute Mechanism

Reasoning models "think" by generating a series of internal, intermediate steps before finalizing a response. This process, which relies on test-time compute, trades speed for accuracy by allowing the model to explore, verify, and refine its solution path. It is a structured deliberation, not random guessing, guided by patterns the model learned during its training.

Here is a step-by-step breakdown of the mechanism:

  1. Prompt Reception: The model receives the user's query just like any other LLM.
  2. Internal Deliberation: Instead of immediately generating the final answer, the model starts creating an internal chain of thought. This takes the form of reasoning tokens (or "thinking tokens"). These tokens are part of the model's output stream but are often hidden from the end-user (Source).
  3. Trajectory Exploration: The model explores different solution paths. For a math problem, it might try one formula, check the result, and then try another. Research on models like DeepSeek-R1 shows that training with reinforcement learning can lead to emergent behaviors like self-reflection and verification. Its predecessor, DeepSeek-R1-Zero, was a notable example, learning these capabilities directly through reinforcement learning without first being fine-tuned on human-labeled reasoning examples (Source).
  4. Consuming the Budget: This deliberation consumes an allocated compute budget. This might be set as a token limit or a general "reasoning-effort" setting. More budget allows for more thinking tokens but directly increases the cost and time you wait for an answer. Model providers work to make this more efficient; for example, OpenAI reported that its o1 model used 60% fewer reasoning tokens on average than its preview version for a given request (Source).
  5. Answer Synthesis: Once the deliberation is complete—or the budget is spent—the model synthesizes its findings into a clean, final answer. The messy intermediate work on its "scratchpad" is typically hidden, leaving only the polished result.

This entire process is an example of moving the performance-versus-cost trade-off from training time to inference time. Instead of trying to bake all knowledge into the model's weights, the model is given the tools and time to "figure things out" on a per-query basis.

What Does Internal Reasoning Look Like in an Example?

To make the concept of test-time compute concrete, consider this classic logic puzzle: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"

Many people—and many direct-answer LLMs—jump to the incorrect answer of $0.10.

Direct-Answer Model Output

A standard model might process the prompt and immediately generate a response.

  • Input: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"
  • Output: "The ball costs $0.10."
  • Illustrative Cost: 25 total tokens, 0.5 seconds latency.

This answer is fast but incorrect. The model defaults to a simple subtraction heuristic ($1.10 - $1.00) without verifying the logic.

Reasoning Model's Conceptual Process

A reasoning model tackles the same problem by first creating an internal monologue.

  • Input: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"
  • Internal Reasoning Tokens (Hidden from user):
    • "Let B be the cost of the bat and C be the cost of the ball."
    • "The problem gives two equations: B + C = 1.10"
    • "And B = C + 1.00"
    • "Substitute the second equation into the first: (C + 1.00) + C = 1.10"
    • "Combine terms: 2C + 1.00 = 1.10"
    • "Subtract 1.00 from both sides: 2C = 0.10"
    • "Divide by 2: C = 0.05"
    • "Let me double check. If the ball is $0.05, the bat is $1.05. Together they cost $0.05 + $1.05 = $1.10. This is correct."
  • Final Output (Visible to user): "The ball costs $0.05."
  • Illustrative Cost: 175 total tokens (150 hidden reasoning tokens + 25 answer tokens), 2.5 seconds latency.

The reasoning model used significantly more tokens and time. However, by breaking the problem down and verifying its work, it arrived at the correct answer. This demonstrates the fundamental trade-off: you pay more in time and compute for a higher chance of accuracy on complex tasks.

How Do Reasoning Models Compare to Alternatives?

Reasoning models are powerful, but they are not the only solution for complex problems. An LLM combined with external tools, often seen in AI agent designs, offers another way to achieve accuracy. The table below compares these approaches.

Feature Direct-Answer LLM Reasoning Model LLM with External Tools (e.g., Code Interpreter)
Best For Simple Q&A, creative text, summarization Multi-step logic, math, coding problems Factual retrieval, precise calculation, execution
Test-Time Compute Minimal High Variable (model compute + tool compute)
Latency Low High Variable (can be high)
Cost (per query) Low High Medium to High
Accuracy (Complex) Low to Medium High High
Verifiability Low (output only) Medium (reasoning can be flawed) High (tool output is verifiable)

What Are the Limits and Common Mistakes?

While powerful, reasoning models are not infallible. Users often make incorrect assumptions about how they work, leading to frustration and misuse. Avoid these common mistakes.

  • Mistake 1: Assuming longer reasoning is always better. Research shows that performance can decline after a task-specific optimal reasoning length. Excessive or repetitive thinking can waste compute without improving the answer (Source).
  • Mistake 2: Trusting the chain of thought implicitly. The generated reasoning is a model's output, shaped by its training to find a correct answer. It is not a guaranteed, faithful record of its computation and can contain errors or post-hoc rationalizations.
  • Mistake 3: Believing they eliminate all errors. Reasoning models can still fail, especially as problem complexity increases. Evaluations show that even strong models struggle when faced with tasks requiring deep relational understanding or when they hit token limits (Source).
  • Mistake 4: Ignoring the cost of hidden tokens. Deliberation is not free. Those hidden "thinking tokens" contribute directly to API costs and response times. An operation that seems simple can become expensive if it triggers a long reasoning chain.
  • Mistake 5: Treating a reasoning-effort setting as a guarantee. While you can influence how much a model "thinks," it is not a precise control. Models may use more or less compute than requested, and more compute does not always yield a better result (Source).
  • Mistake 6: Evaluating only on final-answer accuracy on easy benchmarks. Proper evaluation requires testing on difficult and adversarial cases and measuring latency, cost, and failure modes—not just whether the final answer is right on simple problems.
  • Mistake 7: Assuming reinforcement learning creates human-like understanding. The training methods improve task performance and produce useful behaviors like self-correction. This does not mean the model "understands" concepts in a human-like way; it demonstrates a learned pattern for solving problems (Source).
  • Mistake 8: Overusing verbose reasoning. Requesting that a model reveal its private chain of thought is not always necessary or helpful. For many tasks, a concise final answer backed by checkable evidence is more useful than a long, potentially flawed internal monologue.
  • Mistake 9: Ignoring context and output limits. Long deliberation can exhaust token budgets. For instance, while OpenAI's o1 has a documented 200,000-token context window and 100,000-token maximum output (Source), DeepSeek-R1's benchmarks were run with a 32,768-token cap per response (Source). Hitting these limits can lead to incomplete reasoning and failed tasks (Source).

When Should You Not Use a Reasoning Model?

The key to using reasoning models effectively is knowing when a simpler, faster tool is the better choice. Their high cost and latency make them unsuitable for many common applications.

You should probably not use a reasoning model in these situations:

  • When latency is critical. For real-time applications like conversational chatbots or the systems in a voice AI agent architecture, the multi-second delay of a reasoning model is often unacceptable. A faster, direct-answer model provides a much better user experience.
  • For simple, single-step tasks. Using a reasoning model for simple classification, summarization, or extracting a single piece of information is inefficient. It’s like using a supercomputer to do basic arithmetic. A smaller, faster model will do the job for a fraction of the cost.
  • When a deterministic tool exists. Don't use an LLM to perform a task that a traditional calculator or code interpreter can do perfectly. The best approach is often to use a standard LLM to identify the need for a tool, then call that tool for a verifiable, accurate result.
  • On a tight budget. The high token consumption of reasoning models makes them significantly more expensive for high-volume tasks. If your application makes thousands of calls per day, the cost difference between a direct model and a reasoning model will be substantial.

What to learn next

  • How Are Large Language Models Trained?: Understand the foundational pretraining and fine-tuning processes that create the base models upon which reasoning capabilities are built.
  • What is an AI Agent?: Explore how language models can be combined with external tools and verification steps, an alternative path to solving complex problems.
  • Transformer Architecture Explained: Go deeper into the core architecture that powers nearly all modern language models, including reasoning models.

Sources and further reading