Image segmentation with Segment Anything and its successors

LEARN · COMPUTER VISION

For years, image segmentation models were trained for specific tasks. If you wanted to segment roads, tumors, vehicles, or people, you typically needed a dedicated dataset with pixel-level annotations and a model trained for that domain.

That changed with Segment Anything (SAM). Instead of predicting masks for a fixed set of classes, SAM introduced promptable segmentation: provide an image plus a prompt—a point, bounding box, or existing mask—and the model predicts one or more segmentation masks without task-specific retraining.

Today, image segmentation with Segment Anything and its successors underpins annotation tools, robotics, image editing, and multimodal AI systems. The original idea remains the same, while newer models improve video understanding, efficiency, and language grounding.

In this guide you’ll learn:

What promptable segmentation is
How SAM works internally
The current SAM ecosystem
How to run a working SAM example in Python
Practical deployment tips
Common limitations
Where the technology is heading

Image segmentation assigns every pixel in an image to a meaningful region.

The major segmentation tasks are:

Task	Output
Semantic segmentation	Every pixel receives a class label
Instance segmentation	Each object instance gets its own mask
Panoptic segmentation	Combines semantic and instance segmentation
Promptable segmentation	The user specifies the object through prompts

Traditional segmentation models predict predefined classes.

Promptable models instead answer questions like:

“Segment this object.”
“Segment everything inside this box.”
“Segment the object near this point.”

That flexibility is what made SAM so influential.

SAM consists of three major components:

Image encoder
Prompt encoder
Mask decoder

The workflow is straightforward:

Encode the image once into a high-dimensional embedding.
Encode the user’s prompt.
Fuse both representations in the mask decoder.
Predict one or more candidate masks together with quality scores.

The key optimization is that the expensive image encoder runs only once. Multiple prompts reuse the cached image embedding, making interactive annotation extremely responsive.

SAM accepts several prompt formats.

Point prompts

Click a location inside an object.

A single positive click is often enough to generate a high-quality mask.

Positive and negative points

Additional clicks refine the prediction.

Positive points indicate regions that belong to the object.
Negative points remove unwanted regions.

Interactive refinement is one of SAM’s biggest strengths.

Bounding boxes

Provide an approximate rectangle around the object.

SAM refines it into a pixel-accurate segmentation mask.

Existing masks

A coarse mask from another model can be refined into a cleaner segmentation.

SAM demonstrated that segmentation could become a general-purpose computer vision capability instead of a collection of narrowly trained models.

Common applications include:

Dataset annotation
Robotics
Autonomous driving research
Medical imaging research
Satellite imagery
Scientific imaging
AR/VR
Image editing

Many modern annotation platforms now use SAM-family models to reduce manual labeling effort.

The ecosystem has evolved rapidly. It’s useful to distinguish Meta’s official models from community-developed alternatives.

Model	Maintainer	Best use case
SAM	Meta	General-purpose promptable image segmentation
SAM 2	Meta	Images and video with stateful memory
MobileSAM	Community	Lightweight edge deployment
EfficientSAM	Community research	Faster inference with lower compute
Grounded-SAM pipelines	Combined models	Language-guided segmentation

SAM

The original SAM remains widely used for image annotation, research, and interactive applications.

Its promptable interface established an entirely new segmentation workflow.

SAM 2

SAM 2 is Meta’s current flagship model.

Major improvements include:

Native video segmentation
Stateful memory across frames
Better temporal consistency
Improved interactive workflows
Higher-quality masks on many challenging sequences

For new video projects, SAM 2 is generally the preferred choice.

One important implementation detail is that SAM 2 uses a different inference workflow from the original SAM, particularly for video and stateful tracking. The runnable example later in this article intentionally demonstrates the original SAM through the current Hugging Face Transformers integration because it provides a compact, self-contained example for still-image segmentation.

MobileSAM

MobileSAM focuses on reducing model size and inference cost.

Advantages include:

Smaller checkpoints
Faster CPU inference
Lower memory requirements

The trade-off is that segmentation quality varies depending on the dataset and object complexity, so benchmark it on your own workload.

EfficientSAM

EfficientSAM explores alternative promptable segmentation architectures optimized for speed.

It is especially useful when:

GPU memory is limited
Throughput matters
Latency is more important than achieving the absolute highest mask quality

Grounded-SAM pipelines

Another major trend is combining language grounding models with SAM.

Instead of manually drawing a box, users can write:

Segment every bicycle.

The pipeline becomes:

A grounding model detects candidate objects from text.
Bounding boxes are generated.
SAM converts those boxes into precise masks.

This enables language-driven segmentation without training a dedicated detector.

Install a PyTorch build that matches your CUDA version, then install the remaining packages.

pip install torch torchvision
pip install transformers pillow matplotlib

The following example demonstrates the original SAM using the current Hugging Face Transformers interface. It segments an object from a single positive point.

Place an image named dog.jpg in the same directory before running the script.

from PIL import Image
import matplotlib.pyplot as plt
import torch

from transformers import SamModel, SamProcessor

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = SamModel.from_pretrained(
    "facebook/sam-vit-base"
).to(device)

processor = SamProcessor.from_pretrained(
    "facebook/sam-vit-base"
)

image = Image.open("dog.jpg").convert("RGB")

# Positive click near the object
input_points = [[[300, 220]]]
input_labels = [[1]]

inputs = processor(
    images=image,
    input_points=input_points,
    input_labels=input_labels,
    return_tensors="pt",
)

inputs = {
    key: value.to(device) if hasattr(value, "to") else value
    for key, value in inputs.items()
}

with torch.inference_mode():
    outputs = model(**inputs)

masks = processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu(),
)

scores = outputs.iou_scores[0, 0]
best_index = scores.argmax()

best_mask = masks[0][0][best_index].numpy()

plt.figure(figsize=(8, 8))
plt.imshow(image)
plt.imshow(best_mask, alpha=0.5)
plt.axis("off")
plt.show()

The same interface also supports:

Multiple positive points
Negative points
Bounding boxes
Batched prompts

Most interactive annotation tools follow roughly the same process:

Load an image.
Compute the image embedding.
Click the desired object.
Inspect the predicted mask.
Add positive clicks where regions are missing.
Add negative clicks to remove unwanted areas.
Export the final mask.

Compared with manually drawing polygons, this workflow can dramatically reduce annotation time.

Prompt quality matters.

Some practical guidelines:

Start near the center

Clicks near object boundaries are more likely to produce ambiguous masks.

Use negative clicks early

If the prediction spills into nearby objects, one or two negative clicks usually correct it faster than adding many positive ones.

Prefer boxes for crowded scenes

Bounding boxes provide stronger localization when several similar objects overlap.

Refine instead of restarting

Small prompt adjustments are usually more effective than generating a completely new prediction.

Segmentation models are computationally intensive, but a few optimizations provide substantial speedups.

Cache image embeddings

The image encoder is the expensive stage.

If users interact repeatedly with the same image, compute the embedding once and reuse it for every prompt.

Batch prompts

Instead of processing every click independently:

Collect multiple prompts.
Run a single inference.
Reduce GPU overhead.

Resize intelligently

Higher resolutions improve boundary precision but increase:

GPU memory usage
Latency
Compute cost

Choose the smallest resolution that preserves the detail your application requires.

Use mixed precision

On supported GPUs, mixed precision often reduces memory usage while improving throughput.

with torch.autocast(device_type="cuda", dtype=torch.float16):
    outputs = model(**inputs)

Separate interactive and offline pipelines

Interactive systems prioritize low latency.

Offline processing prioritizes throughput.

Optimizing each pipeline independently usually produces the best overall performance.

Despite impressive zero-shot capabilities, SAM is not perfect.

Common failure cases include:

Tiny objects
Transparent materials
Heavy motion blur
Severe occlusion
Extremely cluttered scenes
Weak object boundaries

Prompt placement also matters. A click in the wrong location can produce an entirely different segmentation.

For production systems, always evaluate on representative data rather than relying solely on benchmark results.

One of the most remarkable ideas behind the original Segment Anything project is that it changed the question from:

“What class is this object?”

“Which object does this prompt refer to?”

That subtle shift transformed segmentation from a fixed-category prediction problem into a general-purpose visual interface.

Even more surprising is the scale that made this possible. The original SAM was trained on SA-1B, a dataset containing approximately 11 million images and 1.1 billion segmentation masks. Instead of memorizing a limited vocabulary of object classes, the model learned a broadly transferable representation that enables strong zero-shot segmentation across many image domains.

That prompting paradigm has since influenced a wide range of multimodal vision systems.

Scenario	Recommended model
Interactive image annotation	SAM or SAM 2
Video segmentation	SAM 2
Mobile deployment	MobileSAM
Memory-constrained inference	EfficientSAM
Language-driven segmentation	Grounded-SAM pipeline

There is no universal winner.

Your choice depends on:

Available hardware
Required latency
Desired mask quality
Whether language prompts are needed
Whether video support is required
Memory constraints

Benchmarking on your own workload is far more valuable than relying exclusively on published benchmarks.

For production systems:

Cache image embeddings whenever possible.
Keep users in the loop for difficult images.
Batch prompts to maximize GPU utilization.
Validate masks before downstream processing.
Benchmark multiple checkpoint sizes.
Measure both latency and segmentation quality.
Test on representative domain-specific images.
Profile preprocessing as well as inference.

Promptable segmentation is becoming a core primitive for multimodal AI rather than a standalone research topic.

Current trends include:

Better language grounding
More robust video understanding
3D scene segmentation
Multi-image reasoning
Improved edge precision
Smaller edge-device models
Tighter integration with multimodal foundation models

Rather than training a different segmentation model for every application, many modern systems now rely on a single promptable backbone that adapts through user interaction or natural-language guidance.

Download a recent SAM-family checkpoint, run the Python example on your own images, and experiment with positive points, negative points, and bounding boxes. Then compare the original SAM with SAM 2 or a lightweight alternative like MobileSAM on your own dataset, measuring mask quality, latency, and annotation speed.

The fastest way to understand modern image segmentation with Segment Anything and its successors is to build with it. Try the code, benchmark multiple models on your own data, and see which approach delivers the best balance of accuracy, speed, and usability for your application.