LLM tokenization is the process that translates human text into a sequence of numbers (tokens) that a model can process. This crucial first step is necessary because LLMs are mathematical systems that operate on numeric IDs, not raw text. Understanding tokenization explains everything from API costs and context limits to why a powerful model can fail at simple tasks like counting letters.

Key takeaways

  • Tokens are not words. A token can be a part of a word, a complete word, punctuation, or a special symbol. Token count is not equivalent to word count (Source name).
  • Tokenization is model-specific. The same text will be tokenized into different tokens and a different number of tokens by different models, such as GPT-4 and Llama 3 (Source name).
  • It is a compression-like process. Techniques like Byte Pair Encoding (BPE) build a vocabulary by merging frequent character pairs. This makes common text more compact but can hide character-level details (Source name).
  • It explains strange model behavior. An LLM's difficulty with tasks like counting letters in a word is a direct result of multiple letters being grouped into a single, indivisible token (Source name).
  • Accurate counting requires code. To know the true token count for API costs and context window limits, you must use the specific tokenizer library for your target model. Simple word or character counting is inaccurate (Source name).

How does LLM tokenization work? A step-by-step guide

Most modern LLMs use a subword tokenization algorithm called Byte Pair Encoding (BPE) or a variant of it. BPE was originally developed as a data compression algorithm (Source name). It "learns" the most efficient way to represent text by creating a vocabulary of common subwords from a large training corpus. The process of applying this learned vocabulary to new text involves several steps.

  1. Normalization: The raw text is first cleaned and standardized. This can involve steps like converting all characters to a specific Unicode format, handling whitespace, or lowercasing text, depending on the tokenizer's design.

  2. Pre-tokenization: The cleaned text is split into an initial set of smaller units. In a common approach called byte-level BPE, the text is first converted into a sequence of raw bytes using the UTF-8 standard. This ensures that any possible string, including any Unicode character or emoji, can be represented and processed (Source name).

  3. Merging: The tokenizer now applies a fixed vocabulary of "merge rules" that it learned during its training. It iteratively scans the sequence and merges the most frequent adjacent pairs into single, new tokens. For example, the bytes for 't' and 'h' might be merged into a 'th' token if that was a common pair in the training data. This process repeats for a fixed number of merges.

  4. ID Mapping: The final sequence of merged tokens is mapped to their unique integer IDs from the model's vocabulary. For example, 'the' might become 464, and ' cat' might become 3767. These IDs are then used to look up embedding vectors, which are the rich numerical representations the model's transformer architecture actually processes.

This multi-step process allows a model to have a finite vocabulary (e.g., 100,000 tokens) but still represent any possible text it might encounter.

A worked example: Byte Pair Encoding (BPE) in action

The BPE merging process is best understood with a simple, illustrative example. Imagine we want to tokenize the string lollapalooza and our tokenizer has learned a small vocabulary with the following merge rules, ranked by priority:

Merge Rules (Priority):

  1. l, l -> ll
  2. o, o -> oo
  3. o, l -> ol
  4. a, p -> ap

Here is how the tokenization would proceed:

Initial State: The string is first split into its basic units (characters in this simplified example). ['l', 'o', 'l', 'l', 'a', 'p', 'a', 'l', 'o', 'o', 'z', 'a']

Step 1: The tokenizer scans for the highest-priority merge, which is l, l -> ll. It finds one instance. ['l', 'o', 'll', 'a', 'p', 'a', 'l', 'o', 'o', 'z', 'a']

Step 2: It scans again. The highest-priority merge rule that can be applied is now o, o -> oo. ['l', 'o', 'll', 'a', 'p', 'a', 'l', 'oo', 'z', 'a']

Step 3: The process repeats. The highest-priority rule applicable to the current sequence is o, l -> ol. ['l', 'ol', 'l', 'a', 'p', 'a', 'l', 'oo', 'z', 'a'] Note: The ll we created in Step 1 broke the adjacent o, l pair, so only the first ol is merged here.

Step 4: The next highest-priority rule is a, p -> ap. ['l', 'ol', 'l', 'ap', 'a', 'l', 'oo', 'z', 'a']

Final State: No more merge rules from our small vocabulary can be applied. The final tokenized sequence is: ['l', 'ol', 'l', 'ap', 'a', 'l', 'oo', 'z', 'a']

The original 12-character string has been converted into 9 tokens. A real tokenizer would have a much larger vocabulary and more complex rules, but the core principle of iteratively merging frequent pairs is the same.

Why do different LLMs tokenize the same text differently?

The same text will produce a different number of tokens and different token boundaries across LLM families. This is because each family trains its own tokenizer with a specific algorithm, vocabulary size, and training dataset. A tokenizer trained on mostly English text will be less efficient for other languages, breaking words into smaller, less meaningful pieces.

Let's compare how different tokenizer families might handle the same complex string. The table below shows an illustrative comparison for the text Tokenization is fun! 🤔 नमस्ते.

Tokenizer Family Visualized Tokens Token Count
GPT-4o [Token][ization][ is][ fun][!][ 🤔][ नम][स्ते] 8
Llama 3 [Token][ization][ is][ fun][!][<0xF0>][<0x9F>][<0xA4>][<0x94>][ नमस्ते] 10
Legacy Model [Token][ization][ is][ fun][!][ ][<unk>][ ][न][म][स][्][त][े] 14

Note: Visualizations are illustrative. Llama 3 may represent unknown characters as individual byte tokens. Legacy models might use an <unk> (unknown) token and fall back to character-level splits.

This variation happens for three main reasons:

  1. Vocabulary: Each model has a unique, fixed vocabulary learned from its training data. The GPT-4o tokenizer has learned नमस्ते as two subwords, while Llama 3's might see it as one.
  2. Algorithm: While most use BPE variants, small implementation differences in normalization and pre-tokenization rules can change the outcome.
  3. Training Data: The efficiency of tokenization for a given language depends heavily on how much of that language was in the tokenizer's training data. This has major implications for efficiency, as the number of tokens needed may vary substantially by script (Source name).

Common misconceptions about tokenization

Several common myths can lead to confusion when working with LLMs. Understanding the reality behind them is key to effective prompt engineering and cost management.

  • Myth: One token always equals one word.
    • Reality: Tokens can be characters, subwords, complete words, punctuation, or special symbols. Token and word counts are different quantities (Source name).
  • Myth: Counting tokens by splitting on spaces is reliable.
    • Reality: This method only produces words, ignoring subwords, punctuation, special tokens, and model-specific rules that determine the actual token count (Source name).
  • Myth: A tokenizer is the same for every LLM.
    • Reality: Token counts depend on the specific model's tokenizer and encoding. Different models will segment identical text differently (Source name).
  • Myth: BPE always finds the mathematically optimal compression.
    • Reality: The underlying optimal pair-encoding problem is computationally difficult (APX-complete). Theoretical analysis provides only worst-case approximation guarantees for the standard greedy BPE procedure (Source name).
  • Myth: One token always equals one character.
    • Reality: Subword tokenizers frequently combine multiple characters into one token. In some cases, a single character might even require multiple byte-derived units (Source name).
  • Myth: A low token count means a prompt has few characters.
    • Reality: Token efficiency varies based on the language, script, whitespace, and encoding. Visible length and token count are not interchangeable metrics (Source name).
  • Myth: Byte-level tokenization makes exact letter counting easy for an LLM.
    • Reality: Byte-level access doesn't solve the core reasoning problem. The model must still align tokenized input with characters and maintain an exact count, which remains a difficult task (Source name).

10 beginner pitfalls to avoid

When starting out, it's easy to make incorrect assumptions about token counting. Here are ten common pitfalls to avoid:

  1. Assuming a fixed words-to-tokens conversion ratio works for every model, language, or script.
  2. Counting spaces or whitespace-separated words instead of using the target model's official tokenizer.
  3. Ignoring how punctuation, line breaks, Unicode characters, emojis, and special tokens are handled.
  4. Using the tokenizer from one model (e.g., Llama 3) to estimate the token count for another (e.g., GPT-4o).
  5. Forgetting that chat roles, message boundaries, tools, files, or other structured data in an API request add to the final token count.
  6. Assuming token boundaries align with linguistically meaningful units like syllables or morphemes.
  7. Treating the token count as a direct measure of semantic meaning, reading difficulty, or the amount of computation the model performs.
  8. Asking an LLM to count its own tokens without verifying the number with an actual tokenizer tool.
  9. Assuming that using a byte-level tokenizer guarantees the model can perform robust character-level arithmetic or letter counting.
  10. Comparing token counts across different tokenizer versions without documenting the model, vocabulary, and specific encoding used for each count.

Key comparisons in tokenization

Understanding tokenization involves distinguishing it from related concepts. Here are four key comparisons.

Word Count vs. Token Count

A word count is based on linguistic definitions (often splitting by spaces), whereas a token count is determined by a model-specific vocabulary, merge rules, and encoding. They are not interchangeable quantities (Source name).

Visible Prompt Text vs. Counted Structured Request

The text you see in a prompt is often only one part of the total input. The final token count in an API call can include additional tokens for roles (e.g., 'user', 'assistant'), message boundaries, and other formatting required by the interface (Source name).

Character Tokenization vs. Subword/Byte-Level BPE

Character tokenization treats every character as a token. This makes character boundaries explicit but results in long sequences. BPE is more efficient, merging frequent units to create shorter sequences, but this process can obscure the internal character boundaries within tokens (Source name).

Greedy BPE vs. Optimal Pair Encoding

Greedy BPE is the practical algorithm used by most tokenizers, which repeatedly chooses the most frequent adjacent pair to merge. Optimal pair encoding, in contrast, is the theoretical problem of finding the absolute best set of merges to achieve maximum compression. This optimal problem is computationally difficult (APX-complete). Theoretical analysis shows the standard greedy procedure has worst-case approximation bounds, with a compression utility between 0.333 and 0.625 of the optimal solution (Source name).

How to accurately count tokens

Because tokenization is model-specific, you cannot rely on character counts, word counts, or online tools that don't specify the model. The only way to get an accurate count is to use the official library for the model you are targeting.

For OpenAI models, you can use the tiktoken library in Python.

Python
import tiktoken

# For gpt-4, gpt-3.5-turbo, text-embedding-ada-002
enc = tiktoken.get_encoding("cl100k_base") 

# For legacy models like davinci
# enc = tiktoken.get_encoding("p50k_base")

text = "Tokenization is fun! 🤔 नमस्ते"
tokens = enc.encode(text)
token_count = len(tokens)

print(f"Text: {text}")
print(f"Token IDs: {tokens}")
print(f"Token Count: {token_count}")

# Example output for cl100k_base:
# Text: Tokenization is fun! 🤔 नमस्ते
# Token IDs: [9906, 1724, 318, 5971, 0, 243, 153, 23, 234, 169, 239, 145]
# Token Count: 12 

Using the correct library is essential for managing costs and ensuring your prompts fit within the model's LLM context window. Remember that for API calls, the final token count may also include special tokens for formatting messages and roles (Source name).

Why do LLMs struggle to count letters?

An LLM's struggle with a seemingly simple task like counting the letters in a word is a direct consequence of tokenization. A model does not "see" the individual characters t-o-k-e-n-i-z-a-t-i-o-n. Instead, it might see two tokens: [Token] and [ization]. The internal character structure is hidden inside these tokens (Source name).

To count the letter 't' in 'tokenization', the model can't just count tokens. It has to perform a multi-step reasoning process: identify the tokens representing the word, "look inside" each token, identify the characters within, check if each character is a 't', and keep a running count. This is a complex algorithmic task, not a native ability.

Research shows that LLMs make counting errors even for short words between 3 and 10 characters long (Source name). These errors are linked to the complexity of the counting operation itself, not just how frequently the model has seen a word (Source name). This reveals a fundamental limitation: what seems easy for humans can be difficult for a model if it contradicts its basic computational units.

What to learn next

Sources and further reading