Deploying vision models to the edge: quantization and latency

Training a computer vision model and deploying that model on a real device are two different engineering disciplines. A network that achieves excellent accuracy in a cloud notebook with a powerful GPU can become impractical when moved to a camera, robot controller, mobile phone, factory inspection system, or battery-powered embedded computer.

The edge changes the optimization target. The question is no longer only “How accurate is the model?” The real question becomes:

• Can the model respond quickly enough?

• Can it fit into available memory?

• Can it operate within thermal and power limits?

• Can it run without a reliable network connection?

• Can the complete application, including preprocessing and post-processing, meet latency requirements?

A prediction that takes 200 milliseconds may be acceptable for a batch analytics system. The same delay may be unacceptable for a robotic system making decisions during motion or a production line inspecting hundreds of items per minute.

Modern edge AI systems require a deployment mindset from the beginning. The model, runtime, hardware accelerator, data pipeline, and security model all influence the final result.

A practical workflow usually looks like this:

1. Train and validate the vision model.

2. Export it into a production runtime format.

3. Optimize the model graph.

4. Apply techniques such as quantization.

5. Benchmark on the actual target hardware.

6. Validate that accuracy remains within acceptable limits.

7. Monitor performance and security after release.

The most important lesson is simple: deployment is not the final step after machine learning. Deployment is part of machine learning engineering.

Most neural networks are trained using 32-bit floating-point values, commonly called float32. These values provide enough numerical precision for gradient-based training and high-quality inference.

The problem is that float32 consumes significant storage and memory bandwidth.

A single float32 value requires four bytes. A model with tens of millions of parameters can therefore require hundreds of megabytes before considering runtime memory, intermediate activations, and framework overhead.

Quantization reduces the precision used to represent model values.

The most common conversion for edge workloads is:

float32 → int8

Instead of storing numbers using 32 bits of floating-point precision, the model stores many values using 8-bit integers.

This can provide:

• approximately four times smaller weight storage

• lower memory bandwidth requirements

• faster execution on hardware optimized for integer arithmetic

• reduced energy consumption

The tradeoff is that lower precision introduces approximation.

Quantization is not a free compression trick. It is a controlled change to the numerical representation of a model.

A simplified conversion looks like this:

quantized_value = round(real_value / scale) + zero_point

The scale and zero point allow the runtime to approximately reconstruct the original numerical range.

For example, a layer containing values such as:

-0.72, -0.15, 0.03, 0.81

might store integer approximations together with metadata describing how those integers map back to floating-point values.

The goal is not to perfectly preserve every decimal place. The goal is to preserve the predictions that matter.

There are three common approaches used in production systems.

Dynamic range quantization

Dynamic range quantization converts weights ahead of time while calculating some activation information during execution.

It is often the easiest first experiment because it requires less preparation.

Advantages:

• simple conversion workflow

• no calibration dataset required

• useful for quickly testing model size reduction

Limitations:

• may not provide the maximum latency improvement

• acceleration depends heavily on the target processor

• some hardware prefers fully integer models

Post-training static quantization

Static quantization converts weights and activations.

This requires a calibration dataset containing representative inputs.

The calibration data should resemble real production data.

For example, a warehouse object detection system should use images showing:

• real camera positions

• expected lighting conditions

• typical object sizes

• normal backgrounds

• realistic image quality

The calibration data usually does not need labels. The runtime mainly needs examples of the numerical ranges it will encounter.

Using random images or unrelated internet photos can create poor calibration results because the model may experience very different activation ranges in production.

Quantization-aware training

Quantization-aware training introduces quantization effects during training or fine-tuning.

Instead of training only with perfect floating-point arithmetic and converting afterward, the model learns to tolerate reduced precision.

A typical workflow is:

1. Train a standard model.

2. Fine-tune with quantization simulation enabled.

3. Export the optimized model.

4. Compare accuracy and latency.

This usually provides the strongest accuracy after conversion, but it requires additional training time and infrastructure.

The examples in this guide target commonly used APIs available in current 2025-era releases:

• PyTorch 2.7.x for model development

• torchvision 0.22.x for vision model utilities

• ONNX 1.18.x for model interchange

• ONNX Runtime 1.22.x for inference and optimization workflows

• TensorFlow 2.19.x and TensorFlow Lite for mobile and embedded deployment

Exact versions should always be pinned in production environments because machine learning frameworks evolve quickly.

A typical development environment can be created with:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install torch==2.7.0 torchvision==0.22.0 onnx==1.18.0 onnxruntime==1.22.0

For a production project, store these dependencies in a lock file or deployment image so that a future package update does not silently change model behavior.

ONNX provides a portable representation for machine learning models. It allows a model trained in one ecosystem to run through compatible runtimes in another environment.

The following example exports a MobileNetV3 model using current PyTorch and torchvision APIs.

import torch
from torchvision.models import mobilenet_v3_small, MobileNet_V3_Small_Weights

model = mobilenet_v3_small(
    weights=MobileNet_V3_Small_Weights.DEFAULT
)

model.eval()

example_input = torch.randn(
    1,
    3,
    224,
    224
)

torch.onnx.export(
    model,
    example_input,
    "mobilenet_v3_small.onnx",
    input_names=["image"],
    output_names=["prediction"],
    opset_version=18
)

print("ONNX export complete")

Export success does not automatically mean production readiness.

Always verify:

• operators are supported by the target runtime

• input shapes are correct

• preprocessing matches training

• outputs match the original framework

Common deployment mistakes include:

• converting RGB images into BGR accidentally

• changing normalization values

• resizing images differently

• swapping channel order

• forgetting model-specific preprocessing

A fast incorrect model is still an incorrect system.

ONNX Runtime provides a production inference engine with support for multiple execution providers depending on the platform.

The runtime may use:

• CPU execution

• GPU execution

• vendor-specific accelerators

• specialized neural processing hardware

A minimal inference example:

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession(
    "mobilenet_v3_small.onnx",
    providers=["CPUExecutionProvider"]
)

input_name = session.get_inputs()[0].name

image = np.random.rand(
    1,
    3,
    224,
    224
).astype(np.float32)

outputs = session.run(
    None,
    {
        input_name: image
    }
)

print(outputs[0].shape)

The random tensor above is only a demonstration. A real application should use the exact preprocessing pipeline from training.

ONNX Runtime includes quantization utilities, but conversion is not guaranteed for every model.

Successful integer conversion depends on:

• supported operators

• model graph structure

• selected execution provider

• available hardware acceleration

A model may require graph changes or operator substitutions before it can be efficiently quantized.

A simplified static quantization example:

import numpy as np
from onnxruntime.quantization import (
    quantize_static,
    CalibrationDataReader,
    QuantType
)


class CalibrationReader(CalibrationDataReader):
    def __init__(self):
        self.samples = iter(
            [
                {
                    "image": np.random.rand(
                        1,
                        3,
                        224,
                        224
                    ).astype(np.float32)
                }
                for _ in range(20)
            ]
        )

    def get_next(self):
        return next(self.samples, None)


quantize_static(
    "mobilenet_v3_small.onnx",
    "mobilenet_v3_small_int8.onnx",
    CalibrationReader(),
    weight_type=QuantType.QInt8
)

print("Quantization complete")

For real deployment, replace the synthetic samples with representative production images.

A synthetic calibration pipeline is useful for testing whether the tooling works, but it does not represent the real data distribution your model will encounter.

TensorFlow Lite remains widely used for mobile and embedded inference.

A basic conversion workflow:

import tensorflow as tf

model = tf.keras.applications.MobileNetV3Small(
    weights="imagenet"
)

converter = tf.lite.TFLiteConverter.from_keras_model(
    model
)

tflite_model = converter.convert()

with open(
    "mobilenet_v3_small.tflite",
    "wb"
) as file:
    file.write(tflite_model)

print("TensorFlow Lite export complete")

For integer quantization, the converter needs representative examples.

import numpy as np
import tensorflow as tf


def representative_dataset():
    for _ in range(100):
        sample = np.random.rand(
            1,
            224,
            224,
            3
        ).astype(np.float32)

        yield [sample]


converter = tf.lite.TFLiteConverter.from_saved_model(
    "saved_model"
)

converter.optimizations = [
    tf.lite.Optimize.DEFAULT
]

converter.representative_dataset = (
    representative_dataset
)

converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]

converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

model = converter.convert()

with open(
    "model_int8.tflite",
    "wb"
) as file:
    file.write(model)

As with ONNX Runtime, the final performance depends on the hardware and supported operators.

A model that is smaller after quantization is not automatically faster. If the target device cannot efficiently execute integer operations, the theoretical advantage may not appear in benchmarks.

Many teams benchmark only the neural network call. That creates misleading numbers.

The real user experience includes:

• image capture

• image decoding

• resizing

• normalization

• inference

• post-processing

• output handling

A realistic benchmark measures the complete pipeline.

Warmup runs are also important. Many runtimes perform initialization during the first inference.

Example benchmark:

import time
import numpy as np
import onnxruntime as ort


session = ort.InferenceSession(
    "mobilenet_v3_small_int8.onnx",
    providers=["CPUExecutionProvider"]
)

input_name = session.get_inputs()[0].name

sample = np.random.rand(
    1,
    3,
    224,
    224
).astype(np.float32)


for _ in range(10):
    session.run(
        None,
        {
            input_name: sample
        }
    )


latencies = []

for _ in range(100):
    start = time.perf_counter()

    session.run(
        None,
        {
            input_name: sample
        }
    )

    end = time.perf_counter()

    latencies.append(
        (end - start) * 1000
    )


print(
    f"Average latency: {np.mean(latencies):.2f} ms"
)

print(
    f"P95 latency: {np.percentile(latencies, 95):.2f} ms"
)

Average latency alone can hide failures.

A system averaging 20 milliseconds may still be unsuitable if occasional requests take 300 milliseconds.

Useful production measurements include:

• average latency

• median latency

• 95th percentile latency

• memory consumption

• CPU usage

• accelerator utilization

• energy usage

• startup time

One of the most interesting developments in edge AI is that a smaller device can sometimes outperform a larger general-purpose computer for a narrow workload.

A desktop CPU may have more raw computing capability, but a dedicated neural processing unit can execute a low-precision model extremely efficiently because it was designed specifically for that workload.

This changes how engineers think about performance.

The fastest deployment is not always the device with the highest theoretical compute. It is often the device where the model representation, runtime, and hardware architecture align.

A compact embedded accelerator running an optimized int8 model can deliver lower latency and lower energy consumption than a much larger machine running the same task inefficiently.

A vision system is not only a machine learning application. It is also an image-processing application.

That means security issues in image libraries can become security issues in the entire deployment.

A notable example is CVE-2023-4863, a heap buffer overflow vulnerability in the WebP image handling code used by libwebp.

The issue became important because WebP support existed across many software ecosystems. Applications that accepted images could potentially expose users or devices to attacks through malformed image files.

The lesson for edge deployments is broader:

• image decoding is part of the attack surface

• model security is only one part of system security

• dependencies need regular updates

• embedded devices need a realistic patch strategy

A camera running an object detector is still a computer processing external data.

Security reviews should include:

• image libraries

• inference runtimes

• operating system packages

• firmware

• communication interfaces

Performance engineering and security engineering should happen together.

Before releasing an optimized vision model, confirm:

• The exported model matches the original model predictions.

• Input preprocessing is identical between training and production.

• Quantization accuracy has been measured.

• The runtime supports all required operators.

• Benchmarks run on the real target hardware.

• Warmup behavior has been considered.

• Memory usage fits device constraints.

• Security updates are possible after deployment.

Optimization should be measured, not assumed.

A smaller file does not automatically create a faster application. A quantized model does not automatically create lower latency. Only real measurements on the final hardware can answer those questions.

Deploying vision models to edge devices requires a complete engineering approach.

The strongest results come from combining:

• portable model formats

• appropriate inference runtimes

• careful quantization

• realistic benchmarking

• accuracy validation

• security planning

Quantization is one of the most valuable tools for reducing model cost, but it works best when treated as part of a larger deployment process.

Take one of your existing vision models, export it to ONNX or TensorFlow Lite, create a benchmark on the target device, test integer quantization, and measure the result. Do not guess which optimization works. Build the workflow that proves it.