The Segment Anything Model, or SAM, is a foundational model for image segmentation. Unlike models trained to find a fixed set of objects like "cars" or "people," SAM can identify the boundaries of an object based on a user's prompt. This makes it a flexible and powerful tool for tasks from creating training data to interactive photo editing.

This guide explains how SAM works, how its different versions compare, and its practical limitations.

Key takeaways

  • What it is: SAM is a model for "promptable segmentation." This means it generates a precise outline, or mask, of an object based on user input like a click or a box. It is not limited to a fixed list of object categories (Segment Anything).
  • How it works: It uses an image encoder to process an image once, a prompt encoder for user inputs, and a fast mask decoder to combine them. This architecture allows for real-time, interactive mask generation.
  • It's not just one model: The original SAM (2023) focuses on images. SAM 2 adds video segmentation (SAM 2: Segment Anything in Images and Videos), and later work like SAM 3 introduces text-based "concept" prompts (SAM 3: Segment Anything with Concepts). Choosing the right version for your task is critical.
  • It's not perfect: While powerful, SAM's performance can degrade on specialized images (like medical scans) and with small, transparent, or overlapping objects. Always validate its output for your specific use case.

What is Promptable Segmentation?

Promptable segmentation is an approach where a model generates an object mask based on interactive user guidance. This allows a single, general-purpose model to segment a vast range of objects without needing retraining for each specific task. This "zero-shot" capability is a key feature of the Segment Anything Model (Segment Anything).

This method differs from traditional semantic segmentation. Semantic segmentation models assign a class label from a predefined list, like 'road' or 'sky', to every pixel in an image. They answer the question, "What is at this location?" for a fixed set of categories.

In contrast, promptable segmentation answers the question, "What is the boundary of the object I am pointing to?" The user provides the context, and the model finds the outline. You can think of it like a digital "magic wand" tool that you guide, whereas semantic segmentation is more like an automatic labeling machine with a fixed set of stickers.

How the Segment Anything Model Works: An Inside Look

The Segment Anything Model uses a three-part architecture to generate masks from prompts efficiently. The design separates the heavy image processing from the lightweight interactive steps, making it fast enough for real-time use. The model was trained on the large-scale SA-1B dataset, which contains over 1 billion masks on 11 million images (Segment Anything).

Here is a step-by-step breakdown of how it operates:

  1. Image Encoder: First, a powerful Vision Transformer (ViT) processes the input image. This encoder converts the image into a detailed numerical representation, called an embedding. This is the most computationally intensive step, but it only needs to be done once per image. The resulting embedding can be reused for many different prompts on that same image.

  2. Prompt Encoder: Simultaneously, a much lighter encoder converts the user's prompt into its own embedding. It can represent different types of prompts, such as the coordinates of positive and negative points, a bounding box, or even a rough mask.

  3. Mask Decoder: A fast, lightweight decoder takes the image embedding and the prompt embedding as input. It uses an attention mechanism to focus on the parts of the image relevant to the prompt. In milliseconds, it predicts potential masks for the indicated object (Segment Anything). The decoder also outputs a confidence score for each mask, estimating how well it matches the object's boundaries.

Because a single point can be ambiguous, SAM often returns multiple valid masks. For instance, a click on a person's shirt might correspond to just the shirt, the person's entire torso, or the whole person. By providing a few options, the model allows the user to select the correct one without needing a perfect initial prompt (Segment Anything).

From a Single Click to a Final Mask: A Worked Example

The true power of the Segment Anything Model comes from its interactive nature. Users can quickly refine masks by providing additional prompts, guiding the model to the precise output they need. This interactive loop is central to promptable segmentation (Segment Anything).

Let's walk through an illustrative example with a picture of a person holding a coffee cup.

1. Initial Prompt (Positive Point) The user wants to segment the coffee cup. They start by placing a single positive point (a click) in the center of the cup. The model might return three candidate masks because the prompt is ambiguous:

  • Mask A: Just the ceramic body of the cup.
  • Mask B: The entire cup, including the handle.
  • Mask C: The cup and part of the person's hand holding it.

2. Refinement Prompt (Negative Point) The user wants the entire cup but not the hand (Mask B). Mask C is incorrect because it includes the hand. To correct this, the user adds a negative point on the part of the mask that covers the hand.

3. Final Output The model takes the original positive point and the new negative point as a combined prompt. It re-evaluates and generates a new set of masks. The top-scoring mask now correctly outlines the entire coffee cup (body and handle) while excluding the hand.

The final output is a binary mask, which is an array of the same size as the image where pixel values are 1 inside the object and 0 outside. This mask can then be used to create a polygon annotation, cut out the object, or serve as a label for training other AI models.

Which SAM Version Should You Use? (SAM vs. SAM 2 vs. SAM 3)

You should choose a version of the Segment Anything Model based on your specific task, as capabilities have expanded over time. The original SAM is for images, SAM 2 handles video, and later research like SAM 3 introduces text-based concept prompting. Using the wrong version can lead to poor results or unsupported features.

Here is a comparison to guide your decision:

Version Primary Use Case Supported Prompt Types Key Feature
SAM (Original) Interactive & automatic segmentation on single images. Points, Boxes, Masks Establishes the core promptable segmentation architecture for images (Segment Anything).
SAM 2 Segmentation and tracking of objects in videos. Points, Boxes, Masks (extended for video) Adds a memory mechanism to maintain object identity across video frames (SAM 2: Segment Anything in Images and Videos).
SAM 3 Identifying and segmenting matching instances of a concept in images or video. Text Phrases, Image Exemplars Segments objects matching descriptions or image exemplars (SAM 3: Segment Anything with Concepts).

For most general-purpose image editing or annotation tasks, the original SAM is the standard choice. If you need to segment an object and follow it through a video sequence, SAM 2 is designed for that purpose. If your goal is to find all objects belonging to a certain category described by text, look to models based on the SAM 3 research.

What are the common mistakes and limits to know about SAM?

The Segment Anything Model is a generalist, which means its broad capabilities come with important limitations. While powerful for many common objects and scenes, its performance is not guaranteed, especially in specialized domains. Knowing its weaknesses helps you avoid common pitfalls and decide when you might need a different tool.

Misinterpreting Quality Scores

SAM provides a confidence score for each mask, but this score reflects the predicted quality of the boundary, not whether it's the correct object. A mask might perfectly outline a shadow or a reflection with a high score, but if you intended to segment the object casting the shadow, the mask is still wrong. Always use human review to verify semantic correctness.

Struggling with 'Non-Natural' Images

SAM was trained on a massive dataset of natural images (Segment Anything). As a result, its performance often drops on images from specialized fields. For example, research has shown that SAM struggles with accuracy in medical imaging applications and satellite-based permafrost mapping without domain-specific adaptation or fine-tuning (Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions, Assessing Segment Anything Model for Permafrost Mapping). Features in these images can be very different from those in everyday photos.

Failing on Challenging Objects

The model can have trouble with certain types of objects:

  • Thin structures: Wires, fences, or tree branches can be difficult to capture accurately.
  • Transparent or reflective objects: Glass, water, or shiny metal can confuse the model.
  • Objects with complex boundaries: Things like fuzzy animals or intricate lace may result in imprecise masks.
  • Heavy occlusion: When objects are mostly hidden behind others, SAM may struggle to guess the full shape.

Confusing a Mask with Recognition

A common mistake is to assume SAM "recognizes" objects. The model is class-agnostic; it segments based on visual boundaries indicated by your prompt. The output is just a group of pixels—a mask—with no associated label like 'dog' or 'cat'. To identify what the object is, you need a separate classification or detection model.

Failing to Define the Task and Metrics

A common pitfall is using SAM without a clear definition of success. Before evaluating a mask, you must define the segmentation target: is it a distinct object (instance), a general category (semantic region), or an arbitrary area? Furthermore, the model's internal quality score is not a substitute for proper evaluation. For any serious application, performance should be measured using task-specific metrics against a labeled validation dataset.

Overlooking Reproducibility

For tasks like creating datasets, reproducibility is critical. A common mistake is failing to save the exact prompts (points, boxes), model version, and any other parameters used to generate a mask. Without this information, it is impossible to reproduce or debug the annotation workflow.

When is SAM Not the Right Tool?

The Segment Anything Model is a versatile tool, but it's not the best solution for every problem. Using it for a task where a specialized model would excel can lead to inefficiency and lower accuracy. Here are some scenarios where an alternative approach is likely better.

  • When you need to classify a fixed set of objects. If your goal is to find all 'pedestrians' and 'vehicles' in street-view imagery, a traditional semantic or instance segmentation model trained on those specific classes will be more direct and often more accurate. SAM is for segmenting a prompted object, not classifying every pixel into a known category.

  • For high-precision scientific or medical measurement. If you need to measure a tumor's volume from a CT scan or calculate deforestation from satellite images, SAM's zero-shot output is generally not reliable enough on its own. These applications demand rigorous, domain-specific validation and often require models fine-tuned on expert-labeled data to ensure medical or legal compliance (Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions).

  • When real-time performance is critical on limited hardware. The original SAM architecture includes a computationally expensive image encoder. While the embedding can be reused for multiple prompts on the same image, this initial processing step can be a bottleneck for applications on edge devices that require real-time segmentation of novel images (Segment Anything).

  • When a narrow, well-defined task allows for a specialized model. For highly uniform tasks, such as segmenting one specific object type in a consistent environment, a smaller model fine-tuned for that domain may be more efficient and accurate than a general-purpose model like SAM. This is a form of domain adaptation, which is often necessary when zero-shot performance is insufficient (Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions).

What to learn next

Sources and further reading