Vision language models (VLMs) allow us to have conversations about images, ask questions about charts, and get descriptions of photos. These systems combine the pattern-recognition strengths of computer vision with the text-generation capabilities of large language models (LLMs). But an LLM doesn't "see" an image like we do.

Understanding the internal process—from pixels to text—is key to using these tools effectively and knowing when to be skeptical of their answers.

Key takeaways

  • A VLM is a system, not a single model. It typically combines three parts: a vision encoder, a projector, and a large language model.
  • The LLM doesn't see pixels. It processes a sequence of numbers, called visual tokens, which represent different patches of the original image.
  • A small but crucial component called a projector acts as a translator. It aligns the numerical representations from the vision model with the language model's "vocabulary."
  • VLMs are often trained in stages. An initial phase teaches the model to align visual concepts with words, and a second phase uses conversational data to teach it how to follow instructions (Source).
  • A major limitation is hallucination—generating text not supported by the image. This is especially common with small details, counting, spatial relationships, and reading text (Source, Source).

What Are Vision Language Models?

Vision language models are AI systems that jointly process images and text to perform tasks like answering questions about a photo or describing a scene. This allows a user to interact with visual content using natural language. For example, you could upload a picture of your refrigerator's contents and ask, "What can I make for dinner with these ingredients?"

This ability to connect language and vision enables applications like creating accessibility tools for visually impaired users, building more intuitive search engines, and allowing robots to understand commands related to their physical environment. But because they can sound so fluent, it's crucial to understand how they work to separate fact from fiction.

How Does an LLM 'See' an Image? The VLM Pipeline Explained

A large language model doesn't directly process pixels. Instead, it receives a special kind of "translation" of the image. A common VLM architecture, like the one introduced in the LLaVA paper, combines three pre-trained components: a vision encoder, an LLM, and a small connector between them. The process works in four distinct steps.

Step 1: Image Encoding

First, the image is divided into a grid of smaller, fixed-size patches. These patches are fed into a vision encoder, which is a separate model trained to understand images (often a Vision Transformer, or ViT). The encoder processes each patch and converts it into a high-dimensional vector—a list of numbers that captures the key features of that patch. The output is not a single description but a sequence of these vectors, one for each patch.

Step 2: Projection

The vision encoder and the LLM were trained separately and have different "internal languages." The numerical vectors from the encoder exist in a different mathematical space from the word embeddings the LLM understands. A small neural network called a projector acts as a translator. It takes the sequence of visual feature vectors from the encoder and maps them into the LLM's embedding space (Source).

Step 3: Sequence Combination

After projection, the visual vectors are now effectively "visual tokens." They are treated just like the text tokens that represent words in a prompt. The model combines the user's text prompt (e.g., "What color is the car?") with these visual tokens into a single, unified sequence (Source). The LLM now has a sequence that contains both language instructions and numerical representations of the image's content.

Step 4: Text Generation

This combined sequence is fed into the LLM. The LLM, often a standard Transformer model, processes the entire sequence. Its attention mechanism allows it to weigh the importance of different tokens—both text and visual—to inform its response. It then generates an answer one token at a time, using the learned associations between the visual features and the words in its vocabulary to form a coherent sentence (Source). The model never looks back at the original pixels; all its visual information comes from the tokens created in Step 1.

A Worked Example: From Pixels to Text Tokens

Let's make this process concrete with a simplified, illustrative example.

Imagine you upload a tiny 64x64 pixel image and ask, "What is in this image?"

  1. Encoding: The vision encoder splits the 64x64 image into a 4x4 grid of 16 patches, each 16x16 pixels. It processes these 16 patches and outputs 16 feature vectors. For this example, let's say each vector has 512 dimensions. The image is now represented by a [16 x 512] matrix.

  2. Projection: Our LLM expects all its input tokens to be represented by vectors of 1024 dimensions. The projector, a simple linear layer, takes each of the 16 vectors and transforms it from 512 dimensions to 1024 dimensions. The visual information is now a [16 x 1024] matrix.

  3. Combination: The user's prompt, "What is in this image?", is tokenized into 6 text tokens. Each of these is also looked up in the LLM's embedding table to get a 1024-dimensional vector. The final input sequence is formed by prepending the visual tokens to the text tokens.

    Code
    <image_token_1> <image_token_2> ... <image_token_16> <What> <is> <in> <this> <image> <?>
  4. Final Input: The LLM receives a single sequence of 22 tokens (16 visual + 6 text). Each token is a vector of 1024 numbers. The model's self-attention mechanism can now process this full sequence to generate an answer, drawing connections between the visual information and the user's question.

How Are Vision Language Models Trained?

Most modern vision language models are not trained from scratch. Instead, they combine vision and language models that have already been pre-trained on vast datasets. The training process focuses on teaching these separate components to work together, typically in two stages (Source).

Stage 1: Feature Alignment

The first stage aims to teach the projector how to translate between the "language" of the vision encoder and the LLM. During this phase, the weights of both the vision encoder and the LLM are usually "frozen," meaning they are not updated. Only the projector is trained.

The model is shown a massive dataset of image-caption pairs. For each pair, the vision encoder creates visual features, and the projector maps them into the LLM's space. The training goal is to adjust the projector so that when the LLM receives these visual tokens, its next-token prediction is the corresponding caption. This aligns visual concepts (like the features of a cat) with word tokens (like "c-a-t").

Stage 2: Visual Instruction Tuning

Once the features are aligned, the model knows how to associate image parts with words. The second stage teaches it to be a helpful assistant. Here, the LLM's weights are "unfrozen" and fine-tuned on a smaller, higher-quality dataset of multimodal conversations. This dataset contains images paired with questions, instructions, and detailed answers.

This process, called visual instruction tuning, teaches the VLM to follow commands, answer questions logically, and conduct a conversation about an image, rather than just providing a simple description (Source). It refines the model's ability to ground its language generation in visual evidence. You can learn more about general LLM training in our guide on How Are Large Language Models Trained? A Complete Guide.

What Are the Common Mistakes and Limits of Vision Language Models?

Vision language models can produce fluent and accurate descriptions, but they are prone to specific types of errors. Understanding these limitations is essential for critically evaluating their outputs. A model's confidence is not a reliable indicator of its accuracy.

Visual Hallucination

The most significant limitation is visual hallucination: generating objects, attributes, or relationships that are not present in the image (Source). The model might confidently describe a person wearing a hat when they are not, or invent details about the background. This can happen when the model relies on language patterns from its training data, a phenomenon known as textual bias, rather than the visual evidence (e.g., "firefighter" is often associated with "helmet"). One 2024 analysis found that over half the models studied had hallucination rates above 40% on a recent benchmark test.

Poor Optical Character Recognition (OCR)

VLMs struggle to read text embedded in images, especially if it is small, stylized, part of a dense document, or seen from an angle. The same study noted that OCR questions were a major source of error, with an average hallucination rate of approximately 60% (Source). The model might misread words, invent text that isn't there, or claim it cannot read text that is clearly visible.

Counting and Spatial Reasoning

Tasks that require precise counting or understanding spatial relationships are also challenging. The patch-based nature of the vision encoder makes it difficult for the model to get a precise global count of many small objects. It may also struggle with prompts like, "Is the red block to the left of the blue sphere?" because this requires fine-grained relational reasoning that the visual tokens may not capture.

When Not to Use Vision Language Models

Given these limitations, you should be cautious when using VLMs for tasks requiring high precision and reliability. Avoid using them as the sole source of truth for:

  • Medical diagnosis from images like X-rays or scans.
  • Automated surveillance where misidentification has serious consequences.
  • Fact-checking claims based on text within an image.
  • Precise measurements or counting for inventory or scientific analysis.
Model Type Primary Function Example Output Key Difference
Image Classifier Assigns an image to one or more predefined labels. {\"label\": \"cat\", \"confidence\": 0.98} Produces structured labels, not free-form text.
Image Captioning Model Generates a single, descriptive sentence for an image. "A black cat is sitting on a windowsill." Optimized for one task: description. Not conversational.
Vision Language Model Answers questions and follows instructions about an image. User: "What color is the cat?" Model: "The cat is black." Supports open-ended dialogue and instruction-following.

What to learn next

Sources and further reading