Object detection today: from YOLO to open-vocabulary models

LEARN · COMPUTER VISION

Object detection sounds simple:

Find objects in an image and tell me where they are.

For years, computer vision systems answered this question using a fixed vocabulary:

“Find cars.”
“Find people.”
“Find bicycles.”
“Find dogs.”

Modern ComputerVision systems are changing the question. Instead of choosing only from categories known during training, newer models can connect visual information with natural language:

“Find the object that matches this description.”

That shift—from closed-set detection to open-vocabulary detection—is one of the biggest changes in ObjectDetection today.

In this guide, we will explore:

How object detectors work
Why YOLO became a real-time computer vision standard
How transformer-based detectors changed the field
What open-vocabulary detection means
How to run current detection models with Python
When to choose traditional detectors versus vision-language systems

1. What object detection actually does

Image classification answers:

What is in this image?

Object detection answers:

What objects are present, where are they, and how confident is the model?

A typical detector returns:

Class label — the predicted object category
Bounding box — the object’s location
Confidence score — the model’s confidence

A detection might look like:

Object: bicycle
Confidence: 0.94
Bounding box: (120, 80, 420, 510)

Bounding boxes are commonly represented as:

x1, y1, x2, y2

where:

x1, y1 are the top-left coordinates
x2, y2 are the bottom-right coordinates

Traditional detectors are trained on labeled datasets where humans define the categories the model should recognize.

That approach works extremely well—but it has a limitation.

2. The YOLO revolution: making detection fast

The YOLO family changed practical computer vision by focusing on speed.

YOLO stands for You Only Look Once. The original idea was that a neural network could process an image in one forward pass and directly predict:

Object locations
Object categories
Confidence scores

This approach helped make real-time detection practical for:

Security cameras
Robotics
Manufacturing inspection
Traffic analysis
Edge AI devices

Current YOLO implementations continue this focus on efficient inference. For example, Ultralytics provides YOLO11 models with a simple Python interface for detection workflows.

Typical YOLO-style detectors are excellent when you already know what you need to detect:

Safety helmets
Cars
Product defects
People
Industrial parts

The model is fast because the problem is well-defined.

3. Running a modern YOLO detector

A current YOLO workflow can be very short.

Install the package:

pip install ultralytics

Run inference:

from ultralytics import YOLO

model = YOLO("yolo11n.pt")

results = model("street.jpg")

for result in results:
    for box in result.boxes:
        class_id = int(box.cls[0])
        confidence = float(box.conf[0])

        label = model.names[class_id]

        print(f"{label}: {confidence:.2f}")

This example:

Loads a pretrained YOLO11 nano model
Runs detection on an image
Prints detected classes and confidence values

The smaller YOLO models are useful when latency matters. Larger models usually trade speed for improved accuracy.

4. The limitation of closed-set detection

A traditional detector only understands the categories it was trained to recognize.

Imagine a model trained with these labels:

person
car
bicycle
dog
chair

It does not automatically know:

electric scooter
unusual tools
rare animal species
specific product models
a “blue backpack with a logo”

The problem is not that the model cannot see the object.

The problem is that the model has no category for it.

Adding new concepts usually requires:

Collecting images
Creating annotations
Training or fine-tuning a model
Deploying a new version

For many applications, this workflow is too slow.

5. Open-vocabulary detection changes the interface

Open-vocabulary detectors replace fixed labels with language prompts.

Instead of asking:

Which known class is this?

they ask:

Does this image region match this text description?

Examples:

“red backpack”
“person wearing a helmet”
“wooden chair”
“small drone”
“rusty metal container”

The detector receives the text at runtime and searches for matching regions.

This is possible because modern systems combine:

Vision encoders
Language encoders
Large-scale image-text training
Transformer-based architectures

The model is not limited to a single list of categories.

6. How vision-language models enabled this shift

A simplified explanation:

An image is converted into a numerical representation.
Text is converted into a numerical representation.
The model compares the two representations.

For example:

Image embedding:
[0.12, -0.44, 0.91, ...]

Text embedding:
[0.10, -0.39, 0.88, ...]

Similarity:
High

The model learns that certain visual patterns correspond to language concepts.

This does not mean the system understands images like a human. It can still fail when:

Objects are tiny
The description is ambiguous
The scene is unusual
The object is outside the model’s experience

However, it creates a powerful capability: using language as a flexible detection interface.

7. Transformers changed object detection

Earlier detection systems relied heavily on convolutional neural networks.

Transformers introduced another approach:

Use attention mechanisms
Model relationships between image regions
Process global context

A major milestone was DETR (Detection Transformer), which showed that object detection could be formulated as a transformer-based prediction problem.

A simplified pipeline:

Image
  |
  v
Feature extractor
  |
  v
Transformer
  |
  v
Object queries
  |
  v
Bounding boxes + labels

This architecture influenced later systems, including open-vocabulary detectors.

8. Running an open-vocabulary detector with Grounding DINO

A practical example is Grounding DINO, an open-set detector that combines DINO-style object detection with language grounding. It can detect objects based on text inputs such as category names or referring expressions.

Install the required libraries:

pip install transformers torch pillow requests

Example inference:

import requests
from PIL import Image
import torch

from transformers import (
    AutoProcessor,
    AutoModelForZeroShotObjectDetection,
)

model_id = "IDEA-Research/grounding-dino-tiny"

processor = AutoProcessor.from_pretrained(model_id)

model = AutoModelForZeroShotObjectDetection.from_pretrained(
    model_id
)

image_url = "https://images.cocodataset.org/val2017/000000039769.jpg"

image = Image.open(
    requests.get(image_url, stream=True).raw
)

text_labels = [
    "cat",
    "remote control",
    "person"
]

inputs = processor(
    images=image,
    text=text_labels,
    return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_grounded_object_detection(
    outputs,
    inputs.input_ids,
    threshold=0.25,
    target_sizes=[image.size[::-1]],
)

for detection in results[0]["boxes"]:
    print(detection)

Unlike a normal YOLO model, the categories are supplied as text.

The detector is not restricted to one predefined class list.

9. Cherry on the cake: a detector that works without COCO training data

One surprising result from Grounding DINO was its ability to perform zero-shot detection on the COCO benchmark.

The model achieved 52.5 AP on COCO zero-shot transfer, meaning it was evaluated on COCO detection without being trained on COCO detection data.

That result illustrates the fundamental change:

Older detectors learned:

“These are the categories I know.”

Open-vocabulary detectors move toward:

“This region matches a concept described in language.”

The model is not magically detecting everything, and it still requires careful evaluation. But the interface is dramatically more flexible.

10. YOLO versus open-vocabulary detectors

Both approaches are useful.

Choose YOLO-style detectors when you need:

Very low latency
Predictable categories
Edge deployment
High-volume inference

Examples:

Counting vehicles
Detecting manufacturing defects
Monitoring safety equipment

Choose open-vocabulary detectors when you need:

Flexible categories
Natural language queries
Rapid experimentation
Unknown object types

Examples:

Searching image collections
Building visual assistants
Creating annotation tools
Exploring robotics environments

A common production strategy is to combine both:

Use open-vocabulary models for discovery and annotation
Use specialized YOLO models for fast production inference

11. Production considerations

A successful detector is more than a model checkpoint.

Important factors include:

Latency

How quickly can predictions be produced?

Critical for:

Robotics
Video systems
Interactive applications

Hardware

Models may run on:

CPUs
GPUs
Mobile processors
Edge accelerators

Data quality

Performance depends on:

Lighting
Camera angle
Object size
Domain differences

Evaluation

Useful metrics include:

Precision
Recall
Mean Average Precision (mAP)

A model that performs well in a benchmark may still require testing in the exact environment where it will be deployed.

12. Where object detection is heading

The future of detection is moving toward systems that combine:

Real-time detectors
Vision-language models
Reasoning systems
Robotics platforms

The output of a future visual system may not stop at:

“There is a chair at these coordinates.”

It may become:

“There is a chair near the desk. It appears damaged and may need inspection.”

Object detection is becoming one component of a broader visual intelligence system.

Final takeaway

Object detection has evolved from fixed-category recognition into a flexible interaction between vision and language.

The major ideas to remember:

YOLO made real-time detection practical.
Transformers introduced more flexible detection architectures.
Open-vocabulary models allow language-driven object discovery.
Modern systems are moving from “recognize known objects” toward “find concepts described by humans.”

Build a small project this week:

Run YOLO11 on your own images.
Try Grounding DINO with custom text prompts.
Compare fixed labels against language-driven detection.

Experimenting with both approaches is the fastest way to understand where computer vision is going next.

1. What object detection actually does

2. The YOLO revolution: making detection fast

3. Running a modern YOLO detector

4. The limitation of closed-set detection

5. Open-vocabulary detection changes the interface

6. How vision-language models enabled this shift

7. Transformers changed object detection

8. Running an open-vocabulary detector with Grounding DINO

9. Cherry on the cake: a detector that works without COCO training data

10. YOLO versus open-vocabulary detectors

Choose YOLO-style detectors when you need:

Choose open-vocabulary detectors when you need:

11. Production considerations

Latency

Hardware

Data quality

Evaluation

12. Where object detection is heading

Final takeaway

Topics Everyone Is Talking About No367

How transformer self-attention actually works, with a runnable toy

Build a minimal RAG pipeline: chunk, embed, retrieve, answer