Understanding the LLM cost per token is essential for building economically viable AI applications. While API pricing pages provide a starting point, the true cost depends on your specific workload, deployment model, and hidden factors that go beyond the headline price. This guide provides a practical framework for calculating your real costs for both API-based and self-hosted models.

Key takeaways

  • LLM costs are calculated differently for third-party APIs versus self-hosted models. API users pay a per-token rate, while self-hosters must cover the total operating cost of their infrastructure.
  • Most providers bill for input tokens (your prompt) and output tokens (the model's response) at different rates (OpenAI API pricing). This makes applications that generate long responses more expensive.
  • The cost of self-hosting is driven by hardware utilization. An idle GPU still costs money, which increases the effective price of every token you generate.
  • The advertised token price is only one component of your total bill. A complete cost model must account for factors like API tool usage, data retrieval for RAG, request retries, and infrastructure overhead (Beyond the LLM Token).

What Is LLM Cost Per Token? (API vs. Self-Hosted)

LLM cost per token is the price to process a single unit of text, typically quoted per one million tokens. For API users, this is the rate charged by the provider. For those self-hosting a model, it is the total operating cost divided by the number of tokens processed, meaning you bear the risk of inefficient or idle hardware.

A token is the basic unit of text that a model processes, created through a process called tokenization. The cost of processing these tokens determines whether an AI application is profitable. There are two primary models for this cost.

  1. API Model: You pay a provider a set price for each token you send (input) and receive (output). The provider manages the hardware and its efficiency. You pay only for what you use.
  2. Self-Hosted Model: You manage the hardware (like GPUs) and software yourself. Your cost per token is your total expenses (hardware, electricity, staff) divided by the total tokens you process. If your system is idle, your effective cost per token increases because your expenses are fixed.

The fundamental difference lies in who is responsible for utilization. With an API, the provider worries about keeping their GPUs busy. When you self-host, that responsibility becomes yours.

Feature API Model Self-Hosted Model
Cost Basis Pay-per-token used Total operating cost (hardware, power, etc.)
Idle Capacity Risk Borne by the provider Borne by you
Primary Driver Price list per million tokens Effective throughput and utilization

How Do You Calculate LLM API Costs? (A Step-by-Step Example)

To calculate the cost of a single API request, you must count input and output tokens separately, find their distinct prices, and sum the costs. The formula is Request Cost = (Input Tokens / 1M × Input Price) + (Output Tokens / 1M × Output Price). Many providers also add fees for features like tool use.

Follow these steps to accurately calculate your API costs for a single request.

A Worked Example

Let's calculate the cost for one API call.

  • Prompt: 2,000 input tokens
  • Response: 500 output tokens
  • Illustrative Pricing:
    • Input: $10.00 per 1 million tokens
    • Output: $50.00 per 1 million tokens

Step 1: Calculate the input cost. Divide your input token count by one million and multiply by the input price. Input Cost = (2,000 / 1,000,000) × $10.00 = 0.002 × $10.00 = $0.02

Step 2: Calculate the output cost. Do the same for your output tokens using the output price. Output Cost = (500 / 1,000,000) × $50.00 = 0.0005 × $50.00 = $0.025

Step 3: Sum the costs. Add the input and output costs together to get the total for the request. Total Cost = $0.02 + $0.025 = $0.045

Step 4: Add other charges. Check if your provider adds other fees. Some services charge for using tools or functions, which would be added to this total (Perplexity AI FAQ). Always refer to the provider's current pricing documentation for exact rates (OpenAI API pricing page).

How Do You Calculate Self-Hosted Inference Costs?

For a self-hosted model, the cost per token is your total operating cost divided by the total tokens successfully processed. A simpler, GPU-focused estimate can be made by dividing the GPU's hourly cost by its effective throughput in tokens per second. Low utilization is the biggest driver of high costs, as you pay for hardware even when it's idle.

Calculating self-hosted costs is more involved than reading a price list. The core idea is to find your cost for a period (like an hour or a month) and divide it by the work done in that period.

The Fundamental Formula

The most accurate formula is also the simplest in concept:

Cost per Token = Total Allocated Operating Cost ÷ Total Successfully Processed Tokens

For example, if you allocate $50,000 in monthly costs to an LLM serving cluster that processes 50 billion tokens, your blended cost is $1 per million tokens (Tokenomics of self-hosted LLMs).

A GPU-Focused Quick Calculation

For a faster, hardware-centric estimate, you can focus on the most expensive component: the GPU. The formula requires careful unit conversion.

Cost per 1M Tokens = (GPU Cost per Hour × 1,000,000) ÷ (Effective Tokens/sec × 3,600 seconds/hour)

Illustrative Example:

  • GPU Cost: $2.00 per hour
  • Effective Throughput: 1,000 tokens per second

Cost per 1M Tokens = ($2.00 × 1,000,000) ÷ (1,000 × 3,600) ≈ $0.56

The key term here is Effective Tokens per Second. This is not a benchmark number from a datasheet. It's the actual, sustained throughput your system achieves on your specific workload, accounting for factors like:

  • Batching: Processing multiple requests at once dramatically increases throughput. Techniques like continuous batching are crucial for efficiency. A serving example shows a jump from 90 tokens/sec for a single request to 4,800 tokens/sec with batching (LLM Inference Cost Optimization).
  • Utilization: A GPU costs money 24/7. If it only processes tokens 50% of the time, your effective cost per token doubles.
  • Latency Requirements: Low-latency applications may require smaller batches, which can reduce overall throughput.
  • Model & Hardware: The specific GPU you choose for inference and the model size directly impact performance.

For a fully-loaded cost, you must also add other operating expenses: electricity, networking, host CPUs and memory, software licenses, and engineering and operations staff salaries.

Why Do Input and Output Tokens Have Different Costs?

Input and output tokens often have different costs because they are processed using computationally distinct methods. Input processing is a highly parallel task where all tokens are handled at once. In contrast, output generation is a sequential, one-by-one process that is less efficient and holds serving resources for a longer time.

This pricing difference stems directly from the transformer architecture that powers most modern LLMs.

  • Input Processing (Prefill): When you send a prompt to the model, it can process all of those tokens simultaneously. The self-attention mechanism can calculate relationships between all tokens in parallel. This is computationally dense but fast.
  • Output Generation (Decoding): To generate a response, the model must produce one token at a time. Each new token depends on all the tokens that came before it. This is an autoregressive process. The model generates token #1, appends it to the sequence, then generates token #2, and so on. This sequential work is often the bottleneck in how LLM inference works and ties up expensive GPU memory for the duration of the generation.

Because output generation is less parallelizable and holds resources longer, providers charge more for it. The exact price ratio between input and output varies by model and provider.

What Hidden Factors Affect the True Cost Per Token?

The final cost of using an LLM is influenced by many factors beyond the advertised per-token price. Your application's workload mix, the model's tokenizer, and efficiency techniques like batching all play a role. Furthermore, the true cost of a successful task must include expenses from tool calls, retries, and data retrieval.

To build a realistic budget, consider this checklist:

  • Workload Mix (Input vs. Output Ratio): An application that summarizes long documents (high input, low output) will have a very different cost profile from a chatbot that writes long stories from short prompts (low input, high output).
  • Tokenizer Differences: The same sentence can be split into a different number of tokens by different models. This means running the same prompt through two models with the same per-token price can result in different costs.
  • Batching and Caching: For self-hosted models, batching requests is critical for reducing costs by increasing GPU utilization (Inference economics of language models). For APIs, some providers offer discounts for reusing parts of a prompt that are already in the cache, but this depends on their specific billing rules (LLM Inference Cost Optimization).
  • Application-Level Costs: The cost per token is not the cost per successful answer. You must also factor in:
    • Tool Calls: Fees for the model calling external functions or APIs.
    • Retrieval: The cost of fetching data in a Retrieval-Augmented Generation (RAG) system.
    • Reasoning Steps: The cost incurred by advanced reasoning models before they produce a final answer.
    • Retries and Errors: The cost of failed requests that still consume tokens but do not produce a useful result.

Common Mistakes When Calculating LLM Costs

Many people miscalculate their LLM costs by making a few common errors. Using a blended price for input and output tokens, confusing benchmark throughput with real-world performance, and ignoring idle capacity are frequent mistakes that lead to inaccurate and overly optimistic cost projections.

Avoid these pitfalls for a more realistic budget:

  • Applying a Blended Rate: Averaging the input and output prices hides the significant cost impact of your application's input-to-output ratio. Always calculate them separately.
  • Ignoring the Denominator: When you see a "cost per token" figure, ask: is that per input token, per output token, or a blend of all tokens? A cost per generated token is a different metric than a cost per total billed tokens (Tokenomics of self-hosted LLMs).
  • Confusing Peak vs. Effective Throughput: When self-hosting, don't use a model's maximum theoretical tokens/second to estimate costs. Measure your actual, sustained throughput on your specific workload and hardware.
  • Forgetting Idle Capacity: A self-hosted GPU costs money 24/7. If your application has low traffic overnight, the cost of that idle time must be spread across the tokens you did generate, increasing their effective price.
  • Excluding Application-Level Costs: The final bill isn't just model tokens. It includes everything needed for a successful user interaction, including retries, guardrails, and tool execution.

When Is Cost Per Token the Wrong Metric?

While useful, cost per token can be a misleading metric when it is the only thing you measure. Focusing on it exclusively can cause you to choose a cheap but low-quality model that ultimately costs more due to higher error rates and the need for more retries. For many applications, a better primary metric is the cost per successful task.

Cost per token is the wrong primary metric in these situations:

  • When Quality Varies Greatly: A cheaper model that gives the wrong answer 30% of the time is more expensive than a pricier model that is correct 99% of the time. The cost of failure, retries, and user frustration often outweighs token price differences.
  • When Latency Is Critical: A model might have a low cost per token but be too slow for a real-time user-facing application. The business cost of a poor user experience can be much higher than the inference cost.
  • For Complex, Multi-Step Tasks: In AI agents that use tools, make multiple model calls, and perform reasoning, the cost of a single token is a small part of the total cost of completing the task. The total cost per successful outcome is more meaningful.
  • During Early-Stage Prototyping: When you are still figuring out your product and prompts, the speed of iteration and model capability are more important than optimizing token costs. Focus on finding a model that can perform the task well first, then optimize for cost later.

A better approach is to define a "successful task" for your application and measure the total, fully-loaded cost to achieve it. This provides a more holistic view of the true economics of your AI feature.

What to learn next

Sources and further reading