ONNX: one model format to run everywhere, from server to browser

LEARN · MULTI-MODEL INFERENCE & SERVING

Modern AI deployment rarely happens in the same environment where a model was trained.

A model may begin life as a PyTorch experiment and later need to run in:

A cloud inference API
A mobile application
A browser
An edge device
A specialized accelerator environment

Training frameworks optimize for flexibility and experimentation. Production inference has different priorities:

Low latency
Smaller deployment footprints
Hardware acceleration
Predictable execution
Cross-platform compatibility

ONNX (Open Neural Network Exchange) solves one of the biggest problems in machine learning deployment: moving a trained model between ecosystems.

Instead of shipping the entire training framework with an application, developers can export a model into the ONNX format and run it with optimized inference engines such as ONNX Runtime.

This guide builds a complete workflow:

Export a PyTorch model to ONNX
Validate the generated model graph
Run inference with ONNX Runtime
Benchmark PyTorch versus ONNX Runtime
Apply graph optimizations
Understand deployment and security considerations

The examples use APIs available in current releases including PyTorch 2.7+, ONNX 1.18+, ONNX Runtime 1.22+, and NumPy 2.x.

Why model portability matters

Machine learning training and inference have different requirements.

A research environment often needs:

Dynamic model changes
Debugging tools
Training utilities
Dataset pipelines
Experiment tracking

A production inference environment usually needs only:

The model
Input and output definitions
An inference engine
Hardware drivers

The typical ONNX workflow looks like this:

Training framework
        |
        v
    ONNX model
        |
        v
 Inference runtime
        |
        v
CPU / GPU / accelerator / browser

An ONNX model contains:

A computation graph
Operators
Tensor shapes and metadata
Model weights

The runtime decides how to execute that graph efficiently on the target hardware.

This separation is the core idea behind “ONNX: one model format to run everywhere.”

What makes ONNX different from framework-specific models?

A framework-specific model is often tightly connected to its original environment.

For example, a PyTorch model might define:

output = torch.relu(layer(input))

The ONNX representation describes the computation instead:

Input
 |
Linear operator
 |
Relu operator
 |
Output

The mathematical operation remains the same, but the execution engine gains freedom to optimize the graph.

The ONNX ecosystem includes:

Export tools from frameworks such as PyTorch
The ONNX model interchange format
ONNX Runtime inference engines
Hardware-specific execution providers

ONNX Runtime can optimize graphs before execution using techniques such as:

Constant folding
Operator fusion
Removing unnecessary graph nodes
Memory planning improvements

The model file remains unchanged while the runtime creates a more efficient execution plan.

Exporting a PyTorch model to ONNX

Install the required packages:

pip install torch>=2.7 onnx>=1.18 onnxruntime>=1.22 onnxscript numpy>=2

Create and export a small neural network:

import torch
import torch.nn as nn


class SimpleClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Linear(256, 10),
        )

    def forward(self, x):
        return self.network(x)


model = SimpleClassifier()
model.eval()

example_input = torch.randn(1, 784)

torch.onnx.export(
    model,
    example_input,
    "classifier.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamo=True,
)

print("Export complete")

The dynamo=True export path uses PyTorch’s newer ONNX exporter workflow. It captures the model graph through PyTorch’s compilation infrastructure instead of relying only on older tracing behavior.

After export, classifier.onnx contains the model computation and weights. The original Python class is no longer required for inference.

Validate the ONNX graph

Before deploying a model, validate it as part of your build pipeline.

import onnx


model = onnx.load("classifier.onnx")

onnx.checker.check_model(model)

print("ONNX model is valid")

Validation catches problems such as:

Invalid graph structure
Missing operators
Corrupted model files
Export mistakes

A model conversion step should be treated like a software build step, not a manual experiment.

Running inference with ONNX Runtime

ONNX Runtime executes the model without importing PyTorch.

import numpy as np
import onnxruntime as ort


session = ort.InferenceSession(
    "classifier.onnx",
    providers=["CPUExecutionProvider"],
)

input_name = session.get_inputs()[0].name

sample = np.random.randn(1, 784).astype(np.float32)

outputs = session.run(
    None,
    {input_name: sample},
)

prediction = outputs[0]

print(prediction.shape)

The deployment package now contains:

The ONNX model file
ONNX Runtime
Required hardware drivers

This separation reduces dependencies and makes it easier to move inference workloads between environments.

Benchmarking PyTorch and ONNX Runtime

ONNX Runtime is often faster than running the original training framework, but the result depends on:

Model architecture
Hardware
Batch size
Execution provider
Runtime settings

A useful benchmark must compare equivalent workloads and include warm-up runs.

This complete benchmark script measures CPU inference latency:

import time

import numpy as np
import onnxruntime as ort
import torch
import torch.nn as nn


class SimpleClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Linear(256, 10),
        )

    def forward(self, x):
        return self.network(x)


model = SimpleClassifier()
model.eval()

example_input = torch.randn(1, 784)

torch.onnx.export(
    model,
    example_input,
    "classifier.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamo=True,
)

session = ort.InferenceSession(
    "classifier.onnx",
    providers=["CPUExecutionProvider"],
)

input_name = session.get_inputs()[0].name

torch_input = torch.randn(1, 784)
numpy_input = torch_input.numpy().astype(np.float32)

for _ in range(100):
    with torch.no_grad():
        model(torch_input)

    session.run(
        None,
        {input_name: numpy_input},
    )

runs = 1000

start = time.perf_counter()

with torch.no_grad():
    for _ in range(runs):
        model(torch_input)

torch_seconds = time.perf_counter() - start

start = time.perf_counter()

for _ in range(runs):
    session.run(
        None,
        {input_name: numpy_input},
    )

onnx_seconds = time.perf_counter() - start

print(f"PyTorch: {torch_seconds:.4f}s")
print(f"ONNX Runtime: {onnx_seconds:.4f}s")

The result is specific to this machine. Production benchmarks should run on the actual deployment hardware.

The hidden performance layer: graph optimization

One of ONNX Runtime’s biggest advantages is graph optimization.

The runtime can rewrite a model graph before execution.

For example:

MatMul
  |
 Add

may become:

Fused MatMul + Add

The output is mathematically equivalent, but fewer operations and less memory movement can improve performance.

Enable aggressive graph optimization:

import onnxruntime as ort


options = ort.SessionOptions()

options.graph_optimization_level = (
    ort.GraphOptimizationLevel.ORT_ENABLE_ALL
)

session = ort.InferenceSession(
    "classifier.onnx",
    sess_options=options,
    providers=["CPUExecutionProvider"],
)

The surprising part is that optimization can improve inference speed without changing model weights.

For example, ONNX Runtime’s transformer optimization work has reported approximately 2x speed improvements for some transformer inference workloads by combining graph transformations and execution improvements. These results are workload-specific and depend on the model architecture, hardware, and runtime configuration.

Running ONNX across different hardware

The ONNX model format stays the same while the runtime chooses how to execute it.

For NVIDIA GPU inference:

import onnxruntime as ort


session = ort.InferenceSession(
    "classifier.onnx",
    providers=[
        "CUDAExecutionProvider",
        "CPUExecutionProvider",
    ],
)

Other execution providers support different hardware environments.

The application code often changes very little because the model representation stays portable.

This is why ONNX is widely used for:

Server inference
Edge AI
Embedded devices
Browser-based machine learning

ONNX in the browser

ONNX Runtime Web allows ONNX models to execute locally in browsers.

Browser-based inference can provide:

Lower latency
Reduced server costs
Improved privacy
Offline functionality

Instead of sending every input to a server, a browser can download a model and execute inference directly on the user’s device.

This approach works best for models that fit within browser memory and performance constraints.

Common ONNX deployment challenges

ONNX improves interoperability, but it does not eliminate every deployment issue.

Unsupported operators

Not every framework operation maps perfectly to ONNX.

Solutions include:

Choosing ONNX-friendly architectures
Replacing unsupported layers
Implementing custom operators

Dynamic input sizes

Some applications require:

Different image sizes
Variable text lengths
Changing batch sizes

Export settings should match the production workload.

Numerical differences

Different execution backends may produce small output differences because they use different kernels and precision strategies.

Always test:

Accuracy
Latency
Memory usage

before production rollout.

Cherry on the cake: ONNX is safer than executable model formats, but the ecosystem still has risks

ONNX files are designed as a graph-and-data format. They are not equivalent to Python pickle-based model files that can execute arbitrary Python objects during loading.

However, the wider machine learning supply chain has suffered from unsafe serialization issues.

A real example is CVE-2024-3568, a vulnerability in Hugging Face Transformers involving unsafe deserialization through pickle.load() in a checkpoint-loading path. An attacker could craft malicious serialized data that executes code when loaded.

The important distinction:

Loading a carefully validated ONNX file is a different security boundary.
Loading arbitrary model artifacts that contain executable serialization logic is much riskier.

Practical safeguards:

Treat downloaded models as untrusted input
Prefer formats without executable deserialization
Verify model provenance
Validate ONNX graphs before deployment
Keep inference libraries updated
Isolate automated model ingestion pipelines

Model files are part of your software supply chain, even when they look like “just data.”

Final thoughts

ONNX is not a replacement for PyTorch. It is the bridge between model development and production inference.

A practical ONNX workflow is:

Train with a flexible framework
Export the model to ONNX
Validate the graph
Enable appropriate optimizations
Run inference with ONNX Runtime
Benchmark on target hardware

The real value of ONNX is interoperability: one model representation can move across servers, devices, browsers, and accelerators.

Build a small neural network, export it, compare inference performance, and experiment with optimization settings. The fastest way to understand ONNX is to run the complete pipeline yourself.