Running models locally: Ollama, llama.cpp, and ONNX Runtime on your own hardware

Running an AI model locally means executing its computations on hardware you control instead of sending every request to a remote inference provider.

Your laptop, desktop, workstation, or private server loads the model, processes inputs, and produces predictions. Once the necessary software and model files are downloaded, many workflows can operate without an internet connection.

This is increasingly practical in 2026. Compact language models can run on ordinary computers, quantization substantially reduces memory requirements, and modern inference tools make deploying models much easier than building a custom inference engine.

Three tools are particularly useful:

• Ollama: A convenient way to download, manage, run, and serve local language models.

• llama.cpp: A configurable inference engine for language models distributed in GGUF format.

• ONNX Runtime: A general-purpose runtime for executing exported machine learning models efficiently.

These tools overlap in some capabilities, but they solve different problems.

In this hands-on tutorial, you’ll learn how to:

1. Estimate the RAM and VRAM required for local inference.

2. Install Ollama and run a compact language model.

3. Query that model through a local HTTP API and Python.

4. Build llama.cpp and run a GGUF model directly.

5. Configure CPU and GPU inference.

6. Train a scikit-learn model using synthetic server telemetry.

7. Export that model to ONNX and perform independent inference.

8. Create a reproducible Python environment using current dependency versions.

9. Benchmark cold-start latency, steady-state throughput, and performance variability.

10. Evaluate security, privacy, licensing, and operating costs.

We’ll use Gemma 3 1B for the first language-model exercises because its modest resource requirements make it a practical starting point. We’ll also explore the newer Qwen 3.5 family.

For the numerical prediction example, we’ll generate synthetic web-server telemetry. No external training dataset is required.

By the end, you’ll have three working local-inference patterns and a framework for deciding which approach belongs in a real application.

1. Understanding the local inference stack

Before installing anything, distinguish three components: the model, its storage format, and its inference runtime.

Confusing these components is a common source of frustration.

The model

A model contains learned parameters, usually called weights, that influence how it processes inputs and produces outputs.

Different models perform different tasks.

Examples include:

• Gemma 3: Google’s model family, including a compact 1-billion-parameter instruction-tuned variant.

• Qwen 3.5: A newer multimodal model family with compact variants such as 0.8B, 2B, and 4B.

• Gemma 4: A 2026-generation family that includes substantially larger models suited to more demanding workloads.

• Linear regression models: Small predictive models that estimate numerical values using learned coefficients.

An LLM generating a paragraph and a regression model predicting server latency both perform inference, but their computational requirements are very different.

The model format

A model format describes how the model’s computation, weights, or associated metadata are stored.

GGUF and ONNX are not interchangeable.

Renaming a GGUF file with an ONNX extension does not convert it into a graph that ONNX Runtime can execute.

Similarly, a scikit-learn regressor exported to ONNX does not automatically become compatible with llama.cpp.

The inference runtime

The runtime performs the computation.

Its responsibilities typically include:

• Loading weights and model metadata.

• Allocating memory.

• Preparing input tensors or tokens.

• Scheduling computations on supported hardware.

• Executing model operations.

• Returning predictions or generated text.

The distinction becomes easier to remember when you associate each tool with its primary purpose.

Ollama manages and serves local language models.

It is designed for developers who want working model inference without configuring every low-level detail.

llama.cpp provides direct control over GGUF-based inference.

It is useful when you want to experiment with quantization, GPU offloading, execution backends, and server configuration.

ONNX Runtime executes compatible exported computation graphs.

It is especially useful for regression, classification, computer vision, and other machine learning workloads where a portable inference artifact is valuable.

What you need before starting

Prepare the following:

• A Windows, macOS, or Linux computer.

• Internet access for initial installations and downloads.

• Several gigabytes of free disk space.

• Approximately 8 GB of system RAM as a comfortable starting point.

• Git, CMake, and a suitable compiler if building llama.cpp from source.

• Python 3.12, 3.13, or 3.14 for the ONNX exercise.

Python 3.14 is the preferred choice for a fresh installation in October 2026, provided the required binary wheels are available for your operating system and architecture.

Python 3.10 is no longer an appropriate default for a new project. Current scikit-learn 1.9 requires Python 3.11 or newer, and the ONNX example will use newer supported Python releases.

The scikit-learn 1.9.1 release and current ONNX Runtime packages document their supported Python versions.

You don’t need a dedicated GPU to complete this tutorial. All three tools support CPU inference for suitable models, although large language models may run substantially slower without hardware acceleration.

2. How much RAM or VRAM do local models need?

One of the biggest misconceptions about local AI is that parameter count directly determines GPU memory requirements.

Parameter count matters, but it is only part of the calculation.

Actual memory consumption depends on:

• The number of model parameters.

• Numerical precision.

• Quantization format.

• Model architecture.

• Runtime buffers and intermediate tensors.

• Context window length.

• Attention-cache implementation.

• GPU offloading configuration.

• Concurrent requests.

A practical memory-sizing formula

A simplified estimate for raw model-weight storage is:

Weight memory in bytes ≈ parameter count × bits per weight ÷ 8

Consider a hypothetical dense model with 8 billion parameters.

At 16-bit precision:

8,000,000,000 × 16 ÷ 8 = 16 GB

At 8-bit precision:

8,000,000,000 × 8 ÷ 8 = 8 GB

At 4-bit precision:

8,000,000,000 × 4 ÷ 8 = 4 GB

These are raw weight-storage estimates, not complete runtime-memory requirements.

Real quantized formats may need additional space for scales, block metadata, and tensors stored at higher precision.

The runtime also requires memory for attention caches, intermediate computations, and other allocations.

What does Q4_K_M mean?

GGUF repositories commonly offer several quantization options.

Quantization reduces the precision used to represent weights.

It can decrease disk usage and memory bandwidth requirements, making inference practical on smaller devices.

However, reducing precision may alter model output quality.

Quantization performance also depends on the model architecture and execution backend. A smaller file isn’t guaranteed to produce proportionally faster inference.

For a concrete example, the ggml-org Gemma 3 1B GGUF repository lists approximately:

• Q4_K_M: 806 MB.

• Q8_0: 1.07 GB.

• F16: 2.01 GB.

Those differences become important when experimenting with limited memory.

Hardware sizing cheat sheet

The following are approximate planning ranges for common dense language models using roughly 4-bit weight quantization.

These aren’t guaranteed hardware requirements. They’re starting estimates for capacity planning.

A useful beginner rule is to budget approximately 25–50% more memory than the weight file occupies for a modest-context, single-user experiment.

That allowance is not sufficient for every architecture or deployment.

Long context windows, large attention caches, and concurrent requests can require substantially more memory.

Why context length matters

The context window controls how much information a model can consider during inference.

A longer context can improve certain document-analysis workflows, but it also introduces resource costs.

Depending on the model architecture and runtime, increasing context length can:

• Increase memory allocation.

• Increase prompt-processing time.

• Reduce concurrency.

• Cause GPU-memory exhaustion.

• Increase end-to-end request latency.

For initial experiments, use approximately 2,048–4,096 tokens.

Increase the context window only after the model works reliably and you have a real use case requiring more input.

RAM versus VRAM

Discrete GPUs generally use dedicated video memory.

When the model cannot fit entirely inside VRAM, a runtime may support partial CPU/GPU execution, but performance can suffer.

Apple Silicon machines use unified memory shared between CPU and GPU workloads. That can make larger models accessible on certain configurations, although usable memory is still limited by operating-system and application requirements.

The key lesson is simple:

Choose a model based on available memory and acceptable response time, not only on parameter count.

3. Ollama quickstart: run your first local language model

Ollama provides one of the simplest entry points into local LLM inference.

It handles model downloading, storage, loading, tokenization, execution, and HTTP serving.

You can install it using the official packages from ollama.com/download.

Step 1: Install Ollama

On macOS or Windows, download the appropriate installer and launch the application.

On Linux, the official installation command is:

curl -fsSL https://ollama.com/install.sh | sh

Review remote installation scripts before executing them, particularly on production machines.

Verify the installation:

ollama --version

On a system where Ollama’s background service is not running, start it in a separate terminal:

ollama serve

Desktop installations generally manage the service automatically, so don’t start a second instance if one already owns the listening port.

Step 2: Download Gemma 3 1B

We’ll use a compact instruction-tuned model.

ollama pull gemma3:1b

The model identifier contains two important components.

• gemma3 identifies the model family.

• 1b selects the approximately 1-billion-parameter variant.

Specifying the variant matters because a model family’s default tag may refer to a different, significantly larger model.

List your downloaded models:

ollama list

The download only needs to happen once unless you intentionally update or remove the model.

Step 3: Run your first prompt

Execute:

ollama run gemma3:1b "Explain local AI inference in two short sentences."

Ollama loads the model, processes the prompt, and generates a response.

You can also start an interactive conversation:

ollama run gemma3:1b

Enter your questions directly in the terminal.

Type /bye to exit the interactive session.

Checkpoint: You now have a working language model running on your own hardware.

The initial model download required network access, but ordinary inference with this downloaded local model does not require submitting the prompt to a hosted model provider.

Step 4: Inspect CPU or GPU execution

Run:

ollama ps

Ollama displays currently loaded models and their processor allocation.

The processor column may indicate:

• Fully GPU-resident execution.

• Fully CPU-resident execution.

• Mixed CPU/GPU execution.

If an available GPU isn’t being used, consult the Ollama GPU documentation and FAQ to check compatibility and configuration.

CPU execution is not necessarily an error. For a small model and occasional use, it may be entirely acceptable.

4. Query Ollama through its local HTTP API

Command-line inference is useful for experimentation, but applications usually need an API.

Ollama runs an HTTP server that normally listens at 127.0.0.1:11434.

Its native generation endpoint is /api/generate.

Step 1: Send an HTTP request

Open another terminal and run:

curl -sS http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma3:1b",
    "prompt": "List three advantages of running AI locally.",
    "stream": false,
    "options": {
      "temperature": 0.2,
      "num_ctx": 2048,
      "num_predict": 128
    }
  }'

The request contains several parameters worth understanding.

We disable streaming because one complete JSON response is easier to inspect during initial development.

With streaming enabled, Ollama can return incremental response objects so an application can display generated content progressively.

The API also provides timing information, including:

• Model-loading duration.

• Prompt-evaluation duration.

• Generated-token count.

• Generation duration.

• Total request duration.

These metrics will become useful when we benchmark performance.

The endpoint and response fields are documented in the official Ollama generation API reference.

Step 2: Call Ollama from Python

You don’t need a third-party SDK to make basic HTTP calls.

Python’s standard library is sufficient.

Create a file named ollama_client.py.

import json
from urllib.request import Request, urlopen

URL = "http://127.0.0.1:11434/api/generate"

payload = {
    "model": "gemma3:1b",
    "prompt": (
        "Write a short checklist for debugging "
        "a slow web application."
    ),
    "stream": False,
    "options": {
        "temperature": 0.2,
        "num_ctx": 2048,
        "num_predict": 160,
    },
}

request = Request(
    URL,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)

with urlopen(request, timeout=180) as response:
    result = json.load(response)

print("Model response:")
print(result["response"])

tokens = result.get("eval_count", 0)
duration_ns = result.get("eval_duration", 0)

if duration_ns > 0:
    tokens_per_second = (
        tokens / (duration_ns / 1_000_000_000)
    )

    print(
        f"\nGeneration throughput: "
        f"{tokens_per_second:.1f} tokens/s"
    )

Run the program:

python ollama_client.py

The script demonstrates a straightforward inference integration:

1. Construct a JSON request.

2. Send it to the local model server.

3. Decode the response.

4. Extract the generated text.

5. Calculate generation throughput.

Notice that the throughput calculation uses generation duration, not total request duration.

That distinction matters.

A model may generate tokens rapidly but still take considerable time to load or process a large input.

For interactive applications, the user’s actual waiting time can be more important than the raw decoding speed.

When is Ollama the right choice?

Ollama is a particularly good fit for:

• Your first local LLM.

• Desktop assistants.

• Internal document summarization.

• Coding experiments.

• Small business automation tools.

• Local retrieval-augmented generation.

• Testing multiple instruction-tuned models.

Its main advantage is convenience.

You can change models without rebuilding the underlying inference integration.

5. llama.cpp quickstart: direct GGUF inference

Ollama is convenient, but sometimes you need more explicit control over how a model is loaded and executed.

That’s where llama.cpp becomes useful.

Despite its name, llama.cpp supports numerous model architectures beyond Meta’s Llama family.

It uses the GGUF format and offers configurable CPU and GPU execution.

Ollama versus llama.cpp

Ollama gives you a higher-level model-management experience.

llama.cpp provides more direct control over:

• GGUF model files.

• Quantization selection.

• Context allocation.

• Hardware backends.

• GPU layer offloading.

• Generation limits.

• Model-server configuration.

• Local and remote model-file acquisition.

The project also provides an HTTP server.

Step 1: Install llama.cpp

On macOS, Homebrew provides a convenient installation option:

brew install llama.cpp

For the following exercises, we’ll build from source on Linux or macOS so that the executable paths are consistent.

First, ensure Git, CMake, and a working C/C++ compiler are installed.

Clone the repository:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

Configure and build the command-line and server executables:

cmake -B build

cmake --build build \
  --config Release \
  --parallel 4 \
  --target llama-cli llama-server

On typical Unix builds, the resulting executables are located in the build/bin directory.

The default build supports CPU inference.

On supported Apple hardware, Metal acceleration is available through the project’s build configuration.

Windows users can also install prebuilt binaries using the official llama.cpp releases or supported package managers.

Step 2: Run the Gemma 3 GGUF model

We’ll use the same underlying instruction-tuned model family as the Ollama exercise.

Run this from the llama.cpp source directory:

./build/bin/llama-cli \
  -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M \
  --ctx-size 2048 \
  --n-predict 128 \
  --single-turn \
  --prompt "Explain model quantization in three sentences."

The -hf argument identifies a model repository on Hugging Face.

llama.cpp downloads and caches the requested GGUF artifact when necessary.

The command uses a real published Q4_K_M variant from the ggml-org model repository.

The main options are:

This approach removes the need to manually download the GGUF file before the first run.

Step 3: Download and load a GGUF file directly

For offline deployment, explicit model files can be preferable.

Create a model directory and download the published artifact:

mkdir -p models

curl -L --fail \
  -o models/gemma-3-1b-it-Q4_K_M.gguf \
  https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

Now load it directly:

./build/bin/llama-cli \
  --model models/gemma-3-1b-it-Q4_K_M.gguf \
  --ctx-size 2048 \
  --n-predict 128 \
  --single-turn \
  --prompt "What advantages does GGUF provide?"

The runtime no longer needs to resolve a model repository during inference.

This is useful when deploying into a controlled environment with an approved collection of model artifacts.

In production, record the file’s cryptographic hash so that you can verify you’re loading the intended artifact.

For example:

sha256sum models/gemma-3-1b-it-Q4_K_M.gguf

On macOS, you can use:

shasum -a 256 models/gemma-3-1b-it-Q4_K_M.gguf

The hash identifies the downloaded bytes.

It does not independently prove that the artifact is trustworthy. You still need a trusted source or independently verified expected digest.

Step 4: Enable NVIDIA GPU acceleration

If your computer has a supported NVIDIA GPU and the necessary CUDA development tools, rebuild llama.cpp with CUDA enabled.

From the llama.cpp source directory:

cmake -B build-cuda -DGGML_CUDA=ON

cmake --build build-cuda \
  --config Release \
  --parallel 4 \
  --target llama-cli llama-server

Run the model with GPU layer offloading:

./build-cuda/bin/llama-cli \
  -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M \
  --n-gpu-layers all \
  --ctx-size 2048 \
  --n-predict 128 \
  --single-turn \
  --prompt "Explain GPU acceleration in plain English."

The --n-gpu-layers all option requests that all possible model layers be placed on the GPU.

This doesn’t guarantee that the model fits in available VRAM.

If inference fails because of insufficient memory, try:

• A smaller model.

• A lower-memory quantization.

• A shorter context.

• Partial GPU offloading.

• Closing other memory-intensive applications.

llama.cpp supports other hardware backends as well. Consult its build documentation for CUDA, Metal, Vulkan, and other supported configurations.

Step 5: Start a local llama.cpp server

llama.cpp includes an HTTP server with OpenAI-compatible API endpoints.

Start it with:

./build/bin/llama-server \
  -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M \
  --alias gemma3-local \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 2048

Leave that terminal running.

Open another terminal and send a chat-completion request:

curl -sS http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma3-local",
    "messages": [
      {
        "role": "user",
        "content": "What is the difference between RAM and VRAM?"
      }
    ],
    "temperature": 0.2,
    "max_tokens": 128,
    "stream": false
  }'

The response contains generated content inside the first completion choice.

The alias gemma3-local allows client applications to use a stable model identifier instead of a file path.

This endpoint is documented in the llama.cpp server reference.

Will Ollama and llama.cpp produce identical answers?

Not necessarily.

Even when both use weights derived from the same source checkpoint, outputs can differ because of:

• Quantization.

• Sampling settings.

• Chat templates.

• Prompt formatting.

• Runtime numerical behavior.

• Randomness.

• Token-generation limits.

For a meaningful comparison, control the variables you can.

Use the same prompt, similar quantization, context size, sampling parameters, and output limits.

Then measure both correctness and performance.

6. ONNX Runtime quickstart: export a scikit-learn model

So far, we’ve focused on language generation.

Now consider a different problem.

Imagine you’re building a performance-monitoring system for a web application.

You want to predict response latency using three measurements:

• Requests per minute.

• Average payload size in kilobytes.

• Number of concurrent workers.

A conventional regression model is a better starting point for this task than a generative language model.

We’ll train the model with scikit-learn, export it to ONNX, and execute it using ONNX Runtime.

This demonstrates how training and inference can use different software environments.

Step 1: Create a reproducible Python environment

Use Python 3.14 for this exercise where supported.

On Linux or macOS, create the project directory:

mkdir local-onnx-demo
cd local-onnx-demo

python3.14 -m venv .venv
source .venv/bin/activate

If Python 3.14 is installed under a different executable name, adjust the first command accordingly.

Python 3.12 or 3.13 are reasonable alternatives if your environment does not yet have compatible Python 3.14 packages.

Windows users can create a virtual environment through the Python launcher and activate it using the script appropriate for PowerShell or Command Prompt.

Verify the interpreter:

python --version

Step 2: Pin the dependencies

Instead of installing unconstrained library versions, create a requirements.txt file.

The following version set uses stable releases available as of October 16, 2026.

numpy==2.5.4
scipy==1.18.1
scikit-learn==1.9.1
skl2onnx==1.20.0
onnx==1.23.2
onnxruntime==1.31.0

These versions correspond to published releases from the relevant projects:

• NumPy 2.5.4

• SciPy 1.18.1

• scikit-learn 1.9.1

• skl2onnx 1.20.0

• ONNX 1.23.2

• ONNX Runtime 1.31.0

Install them:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip check

Package installation depends on the availability of compatible wheels for your Python version, processor architecture, and operating system. If your platform lacks a supported binary wheel, choose a supported Python environment rather than assuming a source build will succeed automatically.

These are explicitly pinned published versions, not a claim that the complete combination has been independently benchmarked on every platform. The export-and-inference checks below serve as a practical compatibility smoke test.

Pinning the direct dependencies improves reproducibility, but it is not a complete dependency lock.

For a managed deployment, capture the entire resolved dependency graph and hashes using your organization’s preferred locking tool.

Why the dependency versions matter

Each package has a distinct responsibility.

The converter and runtime are especially important.

A model may train successfully in scikit-learn but fail during conversion if its estimator or preprocessing operations are unsupported.

Similarly, a valid ONNX graph may require operators unavailable in a particular deployment runtime.

The safest approach is to validate conversion and execution as part of your build process.

Step 3: Generate synthetic server telemetry

We’ll create 600 synthetic measurements.

The input features are:

1. Requests per minute.

2. Average request payload size.

3. Concurrent workers.

Our simulated response latency will increase as load and payload size increase.

We’ll add random noise so the training data resembles real measurements rather than a perfectly deterministic formula.

The data-generation process uses a fixed random seed to make the results reproducible.

Step 4: Train and export the regression model

Create train_export.py.

from pathlib import Path

import numpy as np
import onnx
import onnxruntime as ort

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from skl2onnx import to_onnx


# Generate reproducible synthetic telemetry.
rng = np.random.default_rng(42)
n_samples = 600

requests_per_minute = rng.uniform(
    10, 600, n_samples
)

payload_kb = rng.uniform(
    1, 200, n_samples
)

concurrent_workers = rng.integers(
    1, 32, n_samples
)

X = np.column_stack(
    [
        requests_per_minute,
        payload_kb,
        concurrent_workers,
    ]
).astype(np.float32)

noise = rng.normal(
    0, 3, n_samples
)

latency_ms = (
    15.0
    + 0.06 * requests_per_minute
    + 0.18 * payload_kb
    + 2.5 * concurrent_workers
    + noise
).astype(np.float32)


# Reserve the last 100 samples for testing.
X_train = X[:500]
X_test = X[500:]

y_train = latency_ms[:500]
y_test = latency_ms[500:]


# Train the regression model.
model = LinearRegression()
model.fit(X_train, y_train)

sklearn_predictions = model.predict(X_test)

mae = mean_absolute_error(
    y_test,
    sklearn_predictions,
)

print(f"scikit-learn MAE: {mae:.3f} ms")


# Export to ONNX.
onnx_model = to_onnx(
    model,
    X_train[:1],
    target_opset=17,
)

onnx.checker.check_model(onnx_model)

model_path = Path("latency_model.onnx")

model_path.write_bytes(
    onnx_model.SerializeToString()
)

print(f"Exported: {model_path.resolve()}")

print("Model opsets:")
for opset in onnx_model.opset_import:
    domain = opset.domain or "ai.onnx"
    print(f"  {domain}: {opset.version}")


# Execute the ONNX graph on the CPU.
session = ort.InferenceSession(
    str(model_path),
    providers=["CPUExecutionProvider"],
)

input_name = session.get_inputs()[0].name

onnx_predictions = session.run(
    None,
    {input_name: X_test},
)[0].reshape(-1)


# Validate numerical agreement.
np.testing.assert_allclose(
    sklearn_predictions,
    onnx_predictions,
    rtol=1e-3,
    atol=1e-2,
)

print("ONNX validation passed.")

for index in range(5):
    print(
        f"Actual: {y_test[index]:.2f} ms | "
        f"Predicted: {onnx_predictions[index]:.2f} ms"
    )

Run the program:

python train_export.py

The script performs five important operations.

First, it creates synthetic measurements using a seeded NumPy random generator.

Second, it trains a scikit-learn linear regression model.

Third, it evaluates the model using mean absolute error.

Fourth, it converts the trained model into an ONNX computation graph.

Finally, it executes the exported graph using ONNX Runtime and compares its predictions with scikit-learn.

If everything works correctly, the numerical agreement assertion passes and the program creates latency_model.onnx.

Why use ONNX opset 17?

This deserves an explanation because the target opset is not simply a version number you should increase whenever a newer ONNX release appears.

An ONNX opset defines versions of the operators used by a model’s computation graph.

A newer opset may introduce capabilities or behavior that older runtimes cannot execute.

For our linear regression example, we don’t need recent operator features.

Opset 17 is deliberately chosen because:

• The converter supports it.

• Current ONNX Runtime releases support it.

• The estimator relies on straightforward numerical operations.

• It provides a conservative compatibility target.

• It makes the example less dependent on newly introduced operators.

The sklearn-onnx 1.20 documentation describes support for newer operator sets, including opset 22.

Therefore, 17 is a deliberate portability choice, not the newest supported opset.

The conversion output may also contain operator-set imports for additional domains, such as ai.onnx.ml. The script prints the actual imports so you can inspect the exported artifact rather than assuming all operators share one version.

In more complex projects, select the target opset based on both the converter’s supported operations and the capabilities of the intended inference runtime.

Always validate the resulting graph with the runtime you actually plan to deploy.

Why use float32?

NumPy frequently defaults to 64-bit floating-point values.

However, many exported machine learning graphs use 32-bit floating-point tensors.

Our dataset explicitly converts its input features to np.float32.

This ensures the converter receives a representative input with the intended data type.

Without this step, you can encounter confusing input-type mismatches between training and deployment.

Step 5: Perform inference without scikit-learn

Now create a second file named infer_onnx.py.

import numpy as np
import onnxruntime as ort


session = ort.InferenceSession(
    "latency_model.onnx",
    providers=["CPUExecutionProvider"],
)

input_info = session.get_inputs()[0]
output_info = session.get_outputs()[0]

print(
    "Input:",
    input_info.name,
    input_info.type,
    input_info.shape,
)

print(
    "Output:",
    output_info.name,
    output_info.shape,
)


# Features:
# requests/minute, payload KB, concurrent workers
sample = np.array(
    [[240.0, 12.0, 8.0]],
    dtype=np.float32,
)

prediction = session.run(
    [output_info.name],
    {input_info.name: sample},
)[0]

predicted_latency = float(
    prediction.reshape(-1)[0]
)

print(
    f"Predicted latency: "
    f"{predicted_latency:.2f} ms"
)

Execute it:

python infer_onnx.py

Notice what this script does not import:

• scikit-learn.

• skl2onnx.

• The original trained estimator.

It only requires NumPy, ONNX Runtime, and the exported model file.

This demonstrates one of the most useful properties of ONNX deployment.

Training and inference can have separate software dependencies.

You can train a model inside a development environment and deploy the exported computation graph to another compatible environment without shipping the entire training stack.

For this example, a separate inference environment only needs the relevant runtime packages.

Step 6: Understand the input contract

Our exported model expects three numerical features in a specific order.

That order is part of the model’s contract.

Suppose your application accidentally swaps payload size and worker count.

The supplied tensor may still have the correct shape and data type.

The runtime may accept it without producing an error.

But the prediction will be semantically incorrect.

This is one of the most dangerous categories of machine learning deployment bugs because the software appears to work.

In production, define an explicit feature contract covering:

• Feature names.

• Feature order.

• Units.

• Data types.

• Input ranges.

• Missing-value handling.

• Preprocessing requirements.

ONNX stores the computation graph, but your application remains responsible for supplying meaningful input data.

When should you choose ONNX Runtime?

ONNX Runtime is particularly useful when:

• You have a trained model that can be exported.

• You want inference without the original training framework.

• You need portable model artifacts.

• You deploy across multiple supported environments.

• You want CPU or hardware-accelerated execution.

• You need predictable input and output contracts.

For a small regression or classification task, ONNX Runtime can be a better architectural fit than a language-model server.

For interactive chat generation, Ollama or llama.cpp will usually provide a simpler starting point.

7. Choosing current models for your hardware

After the basic exercises, the next challenge is deciding which model to use.

A common mistake is selecting the largest model that technically fits in memory.

That often produces disappointing results because a large model may be too slow for the actual application.

The better starting strategy is to use the smallest model that meets your quality requirements.

Try Qwen 3.5

As of October 2026, Qwen 3.5 provides compact local variants alongside larger models.

The Ollama Qwen 3.5 library includes 0.8B, 2B, 4B, 9B, and larger variants.

Download the 0.8B model:

ollama pull qwen3.5:0.8b

Query it:

ollama run qwen3.5:0.8b \
  "Explain why a database index can improve query performance."

This gives you a second model family to compare with Gemma 3 1B.

Don’t assume a smaller parameter count guarantees a smaller model download.

Packaging, tensor precision, and model architecture can make file sizes differ substantially.

Try Qwen 3.5 through llama.cpp

The ggml-org GGUF repository includes a Q4_0 variant for Qwen 3.5 0.8B.

From the llama.cpp source directory:

./build/bin/llama-cli \
  -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 \
  --ctx-size 2048 \
  --n-predict 128 \
  --single-turn \
  --prompt "Explain database indexing to a beginner."

The Qwen 3.5 0.8B GGUF model repository lists its available files and quantization variants.

You can now compare:

• Gemma 3 1B through Ollama.

• Gemma 3 1B through llama.cpp.

• Qwen 3.5 0.8B through Ollama.

• Qwen 3.5 0.8B through llama.cpp.

This separates comparisons between model families from comparisons between inference engines.

Select models by task

For example, if a regression model can estimate latency accurately in milliseconds, using a language model for every numerical prediction may add unnecessary computation and variability.

Likewise, if your application requires predictable JSON output, don’t rely exclusively on instructions in a prompt.

Use constrained output mechanisms when supported and validate the generated structure before using it.

Remember model licensing

A model being available for download does not automatically grant unrestricted commercial-use or redistribution rights.

Before deployment, inspect:

• The model license.

• Commercial-use permissions.

• Redistribution conditions.

• Model-specific acceptable-use restrictions.

• Attribution requirements.

• Obligations associated with derived or quantized weights.

The Gemma and Qwen families have different licensing arrangements.

Running a model locally changes where inference happens. It does not remove the legal conditions attached to model distribution and use.

8. Benchmarking local inference correctly

A model that generates 100 tokens per second isn’t necessarily faster for your application than one generating 50 tokens per second.

Why?

Because the total user experience includes more than generation throughput.

A useful benchmark separates several stages.

Cold-start latency

Cold-start latency includes loading the model into memory.

A cold request can be considerably slower than later requests, especially when weights must be loaded from disk.

For applications that run frequently, keeping a model loaded can improve responsiveness.

For infrequent workloads, unloading the model can free significant memory.

Time to first token

Time to first token measures the delay before the first generated output becomes available.

It includes work such as model loading and prompt processing, depending on the state of the runtime.

This metric is particularly important for interactive applications.

A model with fast decoding may still feel slow if it takes several seconds to produce the first token.

Generation throughput

Generation throughput is usually expressed in tokens per second.

The formula is:

Generation throughput = generated tokens ÷ generation duration in seconds

Ollama returns generation durations in nanoseconds.

Convert those values to seconds before calculating throughput.

End-to-end latency

End-to-end latency measures the time between sending a request and receiving the completed response.

It includes relevant portions of:

• Transport and request handling.

• Model loading.

• Prompt processing.

• Token generation.

• Response transfer.

• Client-side parsing.

This is often the most meaningful application-level metric.

Why one benchmark result is insufficient

Single measurements are affected by:

• Operating-system scheduling.

• Background applications.

• GPU utilization.

• Temperature and power management.

• Model-loading state.

• Prompt caching.

• Memory pressure.

• Concurrent requests.

Four requests can demonstrate warm-up behavior, but they don’t provide enough evidence for a reliable performance comparison.

Let’s improve the experiment.

A more rigorous Ollama benchmark

We’ll collect:

• One explicitly prepared cold-start request.

• Twelve steady-state requests.

• Median warm-run latency.

• Minimum and maximum warm-run latency.

• Standard deviation.

• Coefficient of variation.

• Median warm-run generation throughput.

First, unload the model.

ollama stop gemma3:1b
ollama ps

Ensure no other application is continuously loading or querying it.

This prepares a cold-start measurement, although operating-system file caching may still make a second cold run faster than the first.

Create benchmark_ollama.py.

import json
import statistics
import time

from urllib.request import Request, urlopen


URL = "http://127.0.0.1:11434/api/generate"
MODEL = "gemma3:1b"

WARM_RUNS = 12

payload = {
    "model": MODEL,
    "prompt": (
        "Describe three ways to reduce database "
        "query latency. Use short sentences."
    ),
    "stream": False,
    "keep_alive": "10m",
    "options": {
        "temperature": 0,
        "num_ctx": 2048,
        "num_predict": 128,
    },
}

body = json.dumps(payload).encode("utf-8")


def run_inference():
    request = Request(
        URL,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    start = time.perf_counter()

    with urlopen(request, timeout=180) as response:
        result = json.load(response)

    wall_seconds = time.perf_counter() - start

    generated_tokens = result.get("eval_count", 0)

    generation_seconds = (
        result.get("eval_duration", 0)
        / 1_000_000_000
    )

    load_seconds = (
        result.get("load_duration", 0)
        / 1_000_000_000
    )

    prompt_seconds = (
        result.get("prompt_eval_duration", 0)
        / 1_000_000_000
    )

    throughput = (
        generated_tokens / generation_seconds
        if generation_seconds > 0
        else 0.0
    )

    return {
        "wall": wall_seconds,
        "load": load_seconds,
        "prompt": prompt_seconds,
        "tokens": generated_tokens,
        "throughput": throughput,
    }


print("Cold-start measurement")
cold = run_inference()

print(f"Wall latency: {cold['wall']:.3f} s")
print(f"Model loading: {cold['load']:.3f} s")
print(f"Prompt evaluation: {cold['prompt']:.3f} s")
print(f"Throughput: {cold['throughput']:.2f} tokens/s")


print("\nSteady-state measurements")

warm_results = []

for index in range(WARM_RUNS):
    result = run_inference()
    warm_results.append(result)

    print(
        f"Run {index + 1:02d}: "
        f"latency={result['wall']:.3f}s, "
        f"speed={result['throughput']:.2f} tokens/s"
    )


latencies = [
    result["wall"]
    for result in warm_results
]

throughputs = [
    result["throughput"]
    for result in warm_results
]

median_latency = statistics.median(latencies)
mean_latency = statistics.mean(latencies)

latency_std = statistics.stdev(latencies)

latency_cv = (
    100 * latency_std / mean_latency
    if mean_latency > 0
    else 0.0
)

median_throughput = statistics.median(throughputs)

throughput_std = statistics.stdev(throughputs)


print("\nSteady-state summary")

print(
    f"Median latency: "
    f"{median_latency:.3f} s"
)

print(
    f"Latency standard deviation: "
    f"{latency_std:.3f} s"
)

print(
    f"Latency coefficient of variation: "
    f"{latency_cv:.2f}%"
)

print(
    f"Latency range: "
    f"{min(latencies):.3f}–"
    f"{max(latencies):.3f} s"
)

print(
    f"Median generation throughput: "
    f"{median_throughput:.2f} tokens/s"
)

print(
    f"Throughput standard deviation: "
    f"{throughput_std:.2f} tokens/s"
)

Execute it:

python benchmark_ollama.py

This provides a much more informative benchmark than printing four individual request timings.

For reference, here is one run on a 12-thread x86-64 laptop CPU with no GPU (Ollama 0.40.5, gemma3:1b, October 16, 2026):

Model loading: 1.295 s
Prompt evaluation: 0.193 s
Throughput: 33.02 tokens/s

Steady-state summary
Median latency: 1.852 s
Latency standard deviation: 0.234 s
Latency coefficient of variation: 12.33%
Latency range: 1.680–2.320 s
Median generation throughput: 30.39 tokens/s
Throughput standard deviation: 3.37 tokens/s

The twelve warm runs ranged from 24 to 33 tokens/s on an otherwise busy machine. That is why the summary reports a median and a spread instead of one number. On the same CPU, llama-cli reported about 31 tokens/s generation for the same Q4_K_M model, which is consistent because Ollama runs a llama.cpp engine underneath.

Interpreting the benchmark

The first request is intended to capture cold-start behavior after the model has been explicitly unloaded.

The subsequent requests measure a warm, loaded model.

The keep_alive setting helps prevent the model from unloading between measurements.

Median latency provides a representative completion time less sensitive to occasional extreme values than a simple mean.

Standard deviation describes the spread of observed latencies.

Coefficient of variation expresses the standard deviation relative to the mean.

For example, a 2% latency coefficient of variation indicates a relatively consistent measurement set, whereas 30% indicates substantially greater variability.

Median warm-run throughput provides a representative decoding-speed estimate without including initial model-loading time.

The benchmark still has limitations.

It sends sequential requests using the same prompt, so prompt caching may influence later measurements. It also measures complete responses rather than streaming time to first token.

Twelve warm runs provide a useful first comparison, not a statistically comprehensive characterization of production performance.

Comparing Ollama and llama.cpp fairly

When benchmarking the two engines, follow the same methodology.

1. Use models derived from the same underlying checkpoint and, where possible, equivalent quantization.

2. Use identical prompts and comparable chat templates.

3. Match context sizes and maximum output lengths.

4. Use equivalent sampling configurations.

5. Keep hardware, power settings, and background workload consistent.

6. Measure cold-start and warm-run behavior separately.

7. Record median latency, throughput, and variability.

8. Inspect actual GPU utilization and memory consumption.

For llama.cpp, its command-line utilities report timing information, while its HTTP server provides request and performance metrics.

Be careful when comparing token rates between different model families. Their tokenizers may encode the same text into different token counts.

A comparison based solely on tokens per second can therefore be misleading.

A fast model can still be the wrong model

Suppose two models produce different results.

These are illustrative figures.

Model A is faster.

But if the application performs important document analysis, Model B may provide much greater practical value.

Accuracy and operational requirements often matter more than raw inference speed.

A useful evaluation should measure:

• Correctness.

• Unsupported or fabricated claims.

• Output-format compliance.

• Response consistency.

• End-to-end latency.

• Memory consumption.

• Resource utilization.

• Error frequency.

Performance optimization should never silently reduce output quality below the application’s acceptance threshold.

9. Privacy and security: local does not automatically mean safe

Running inference on your own hardware can reduce dependence on external data processors.

But local execution should not be confused with comprehensive security.

A local model can still be exposed through insecure network configuration, compromised dependencies, application logs, or overly permissive retrieval systems.

Keep inference APIs private

Ollama normally binds its API to the loopback interface.

Our llama.cpp server example explicitly binds to 127.0.0.1.

This limits direct access to applications running on the same machine.

Avoid exposing an unauthenticated inference endpoint to public networks.

If other devices need access, plan for:

• Authentication.

• Authorization.

• TLS.

• Firewall rules.

• Network segmentation.

• Rate limits.

• Resource quotas.

• Logging and auditing.

A model server can consume substantial memory and processing capacity.

Even when no sensitive data is exposed, unrestricted access can create availability problems.

Disable cloud features when strictly local inference is required

Modern Ollama installations support optional cloud features in addition to local inference.

For a deployment policy that requires local-only operation, Ollama documents a configuration setting that disables cloud functionality.

Create or edit ~/.ollama/server.json.

{
  "disable_ollama_cloud": true
}

Restart Ollama after changing the configuration.

The official Ollama FAQ also documents the equivalent OLLAMA_NO_CLOUD environment variable.

Disabling cloud features is useful, but it does not guarantee that every application, logging agent, or operating-system component is isolated from the network.

A strict air-gapped deployment requires appropriate system and network controls.

Treat model artifacts as software dependencies

Model files should be included in your software supply-chain management process.

For production deployments:

• Download models from approved repositories.

• Record artifact hashes.

• Track licenses and redistribution conditions.

• Pin runtime versions or commit identifiers.

• Review model and dependency updates.

• Test updated inference engines before deployment.

• Restrict permissions on model-storage directories.

Avoid assuming that every model repository is trustworthy simply because its files can be downloaded.

Model data, runtime parsers, and supporting tools all form part of the attack surface.

Protect retrieved documents

Consider an internal knowledge assistant.

Even if its language model runs locally, the application may retrieve documents from a database.

If the retrieval layer does not enforce access permissions, the model could receive information the requesting user is not authorized to view.

The correct architecture enforces authorization before supplying retrieved content to the model.

Prompt injection is another concern.

Instructions contained inside external documents must not be allowed to override application security rules or trigger unauthorized actions.

Local inference provides architectural control, but security still depends on the surrounding software.

10. Cherry on the cake: a real engineering team processing millions of tokens locally

The practical value of local inference becomes clearer when you examine a real production deployment.

On August 3, 2026, the engineering team at theBlue.ai published an account of three locally hosted language-model systems used in its business operations.

The original engineering report is available here:

Local LLMs in Practice: How We Use Them at theBlue.ai — August 3, 2026

The report is particularly interesting because the team didn’t choose local models for a single reason.

Different workloads justified local inference in different ways.

System 1: Procurement opportunity analysis

The team uses a locally deployed Gemma 4 26B model to process public procurement documentation.

According to the engineering report, the system processes approximately 15–20 million tokens per day.

Its workflow includes:

• Identifying relevant procurement opportunities.

• Evaluating eligibility.

• Reading supporting documentation.

• Extracting requirements and deadlines.

• Producing structured recommendations.

The report describes a deployment achieving approximately 88 tokens per second in its particular configuration.

That figure belongs to the team’s hardware, model, and serving setup. It should not be interpreted as a general Gemma 4 performance guarantee.

System 2: High-volume background monitoring

Another system uses a larger Gemma 4 31B model for policy-related monitoring.

The team reports approximately 28 million tokens per day in background processing.

This workload has a different objective.

Users are not necessarily waiting for every generated token in an interactive interface.

That gives the engineers more freedom to prioritize analysis quality over immediate response latency.

System 3: Internal knowledge retrieval

The team also operates an internal document-search assistant using Qwen3 30B.

Its architecture combines retrieval-augmented generation with locally hosted model inference.

The purpose is not only to reduce recurring API calls.

Keeping retrieved company documentation on infrastructure controlled by the organization is also an important design objective.

What makes this story interesting?

Three different workloads led to three different engineering decisions.

The team also describes practical optimization work involving quantization, serving configuration, and architectural techniques.

The broader lesson is that local inference is an infrastructure decision, not a model popularity contest.

The best model and deployment strategy depend on whether the application needs:

• High sustained throughput.

• Low interactive latency.

• Strict data control.

• Specialized output quality.

• Predictable resource allocation.

The same engineering report acknowledges the limitations of local hosting, including hardware investment, operational responsibility, and quality differences compared with some commercial hosted models.

How do you know whether going local saves money?

You need to compare total costs under equivalent workload and quality requirements.

Consider a hypothetical team processing 50 million billable tokens per month.

Assume an illustrative effective hosted inference price of $3 per million tokens for its particular input/output mix.

Its monthly API cost would be:

50 × $3 = $150

Now suppose the local deployment has these allocated monthly expenses.

Under these assumptions, the two approaches have similar monthly costs.

But change the workload and the economics shift.

At 10 million tokens per month, the hypothetical hosted cost becomes $30.

At 200 million tokens per month, it becomes $600.

If the local hardware can handle the additional volume without new infrastructure, local inference becomes more attractive at higher utilization.

If the workload requires expensive new GPUs, additional staff time, or higher availability, the conclusion may change again.

These numbers are illustrative, not actual provider quotations.

A serious cost comparison must include:

• Input and output token pricing.

• Caching discounts.

• Hardware acquisition.

• Depreciation.

• Electricity.

• Cooling.

• Administration.

• Model updates.

• Monitoring.

• Backup capacity.

• Reliability requirements.

• Output quality.

Local inference does not eliminate operating costs.

It replaces a usage-priced inference service with infrastructure and operational expenses.

That trade-off can be excellent for some workloads and unnecessary for others.

11. Troubleshooting common local inference errors

The following problems account for many beginner difficulties.

Ollama reports that a model cannot be found

Check the local model list:

ollama list

If the expected model is missing, pull the exact variant:

ollama pull gemma3:1b

Be precise about model tags.

A model family name does not necessarily identify the same weights or size as a variant-specific tag.

The Ollama API refuses connections

Check whether Ollama is running.

On an installation without an active background service:

ollama serve

Do not start another copy if the service is already running.

The default local API port in our exercise is 11434.

The llama.cpp server example uses 8080.

Confusing the two ports is a straightforward way to create connection errors.

llama.cpp cannot download a model

Verify that:

• The repository exists.

• The requested quantization is published.

• The machine has internet connectivity.

• There is sufficient disk space.

• Any required repository permissions are satisfied.

For controlled environments, downloading a specific GGUF artifact separately may simplify troubleshooting.

GPU acceleration doesn’t work

With Ollama, inspect the loaded model allocation:

ollama ps

With llama.cpp, inspect its startup output and GPU offloading settings.

If the model is unexpectedly running on the CPU:

• Confirm that the hardware backend is supported.

• Check the GPU driver.

• Check the selected runtime build.

• Verify available VRAM.

• Confirm that GPU offloading is configured.

Don’t assume every performance problem is caused by a missing GPU.

An oversized context, heavy background workload, or inappropriate model size can produce similar symptoms.

ONNX Runtime rejects an input tensor

The most common causes are incorrect shapes and data types.

Create a short inspection script named inspect_onnx.py.

import onnxruntime as ort

session = ort.InferenceSession(
    "latency_model.onnx",
    providers=["CPUExecutionProvider"],
)

print("Inputs:")

for item in session.get_inputs():
    print(
        item.name,
        item.type,
        item.shape,
    )

print("\nOutputs:")

for item in session.get_outputs():
    print(
        item.name,
        item.type,
        item.shape,
    )

Run it:

python inspect_onnx.py

Compare the declared input contract with the arrays your application supplies.

If the model expects float32, passing float64 data can cause inference to fail.

If it expects three features, supplying a tensor with four features is invalid.

ONNX predictions differ from scikit-learn

Small numerical differences can result from floating-point precision and implementation details.

Avoid comparing floating-point predictions with exact equality.

Use tolerances appropriate to your application.

The earlier example uses:

np.testing.assert_allclose(
    sklearn_predictions,
    onnx_predictions,
    rtol=1e-3,
    atol=1e-2,
)

Large discrepancies, however, should be investigated.

Check:

• Feature order.

• Input data types.

• Preprocessing consistency.

• Converter support.

• Operator-set compatibility.

• Model-export configuration.

A successful export is only one part of deployment validation.

12. Moving from a tutorial to a reliable deployment

Running a model successfully once is a useful milestone.

Running it reliably for users requires more deliberate engineering.

Pin software and model versions

A production deployment should record:

• The exact model artifact.

• Artifact checksum.

• Model license.

• Inference runtime version.

• Python dependencies, where relevant.

• Quantization configuration.

• Context length.

• Generation parameters.

• Hardware backend.

This allows you to reproduce the deployment if a future update changes behavior.

Avoid relying exclusively on mutable model names.

Where possible, deploy approved immutable artifacts.

Build explicit request validation

Never assume every input is valid.

For language-model applications, validate:

• Request size.

• Maximum input length.

• Allowed output formats.

• Resource limits.

• User authorization.

For ONNX-based numerical prediction, validate:

• Feature names.

• Feature order.

• Numerical ranges.

• Data types.

• Tensor shapes.

• Missing values.

These validations are part of the application contract.

Plan for concurrency

A local inference server does not have unlimited capacity.

Concurrent requests compete for compute, memory, and potentially attention-cache allocations.

A deployment needs a strategy for:

• Request queues.

• Admission control.

• Maximum concurrency.

• Timeouts.

• Overload handling.

• Model-loading failures.

• Recovery after runtime errors.

A system that performs well with one user can behave very differently with twenty simultaneous users.

Measure realistic concurrent traffic before declaring the deployment production-ready.

Keep evaluation separate from serving

A useful project structure might look like this:

local-ai-project/
├── app/
│   ├── api.py
│   ├── inference.py
│   └── validation.py
├── models/
│   └── latency_model.onnx
├── tests/
│   ├── test_inputs.py
│   ├── test_predictions.py
│   └── test_model_quality.py
├── benchmarks/
│   └── benchmark_ollama.py
├── requirements.txt
└── README.md

This is an illustrative structure, not a complete runnable application.

The separation is intentional.

Inference code should be responsible for loading and executing models.

Validation code should enforce the application’s input contract.

Tests should determine whether behavior remains acceptable.

Benchmarks should measure performance independently of normal request handling.

Validate quality before upgrading models

When replacing a model version, don’t assume newer automatically means better.

A replacement model may improve general reasoning while becoming worse at your specific task.

Build a representative evaluation set.

For a retail-support assistant, for example, that might include:

• Shipping questions.

• Return-policy explanations.

• Product comparisons.

• Order-status messages.

• Requests requiring structured responses.

Evaluate the original and replacement models against the same examples.

Record correctness, latency, output validity, and failure cases.

For numerical models, compare held-out error metrics and exported inference results.

Model upgrades should be deliberate changes with measurable acceptance criteria.

Production readiness checklist

Before serving actual workloads, confirm that:

• Model artifacts and versions are recorded.
• Licenses have been reviewed.
• Dependencies are locked or constrained.
• Inputs are validated.
• Network access is restricted appropriately.
• Authentication is configured where required.
• Sensitive prompts and outputs are handled safely.
• Concurrency and resource limits are enforced.
• Model quality is evaluated on representative examples.
• Cold-start and warm-run performance are measured.
• Runtime updates are tested before rollout.
• Failure and recovery procedures are documented.

These practices usually contribute more to long-term reliability than a small improvement in decoding throughput.

13. Which tool should you actually use?

After completing the exercises, the distinctions between the three runtimes should be clearer.

Choose Ollama when convenience matters most

Ollama is a good default when you want to:

• Download an LLM.

• Start a conversation.

• Build a desktop assistant.

• Integrate text generation into a small application.

• Compare several local models quickly.

Its model-management and serving experience reduces setup complexity.

Choose llama.cpp when runtime control matters

Use llama.cpp when you need:

• Direct GGUF loading.

• Explicit quantization selection.

• Fine-grained GPU offloading.

• Hardware-backend experimentation.

• Custom inference configuration.

• Greater control over a lightweight local serving stack.

It’s especially useful for performance tuning and tightly controlled model deployments.

Choose ONNX Runtime for exported machine learning models

Use ONNX Runtime when you have a compatible model and need:

• Inference separated from training.

• Portable model computation.

• A consistent prediction interface.

• CPU or supported hardware acceleration.

• Deployment without the original training framework.

A numerical prediction endpoint is a natural example.

Can you combine them?

Absolutely.

Consider an e-commerce support application.

A small ONNX classifier could categorize incoming requests.

Simple requests might be answered with fixed application logic.

Requests requiring a natural-language explanation could then be routed to an LLM running in Ollama or llama.cpp.

This architecture can reduce computational waste by using each model for the operation it handles best.

There is no requirement to use one inference engine for every machine learning task.

14. Your next hands-on challenge

You now have three concrete approaches to local inference:

• Managed language-model execution through Ollama.

• Direct GGUF inference through llama.cpp.

• Portable numerical-model inference through ONNX Runtime.

The next step is to measure what works on your own hardware.

Challenge 1: Compare two language-model runtimes

Run Gemma 3 1B through Ollama and llama.cpp.

Use a fixed collection of five representative prompts.

Record:

• Response correctness.

• Median warm-run latency.

• Generation throughput.

• Performance variability.

• Memory consumption.

• CPU or GPU utilization.

Keep model configurations as comparable as possible.

Challenge 2: Try a newer compact model

Run Qwen 3.5 0.8B through both supported workflows.

Compare its outputs with Gemma 3 1B.

Don’t rank the models solely by parameter count or throughput.

Decide which performs better on the tasks you actually care about.

Challenge 3: Export a different estimator to ONNX

Replace linear regression with another supported scikit-learn estimator.

For example, train a regression model on synthetic energy-consumption or server-performance data.

Export it to ONNX.

Then verify:

• The exported model loads.

• Its input contract is correct.

• Its predictions match scikit-learn within appropriate tolerances.

• The inference script runs without importing scikit-learn.

Challenge 4: Make an evidence-based deployment decision

Choose three hypothetical applications:

• A lightweight desktop writing assistant.

• A batch document-analysis service.

• A numerical prediction API.

For each one, identify:

• The most suitable model type.

• The best inference runtime.

• The memory requirements.

• The expected latency.

• The security considerations.

• The likely cost trade-offs.

You should find that no single runtime wins every comparison.

A compact language model may be perfect for a desktop assistant but unnecessarily complicated for a latency-prediction endpoint.

A large GPU-hosted model may make financial sense for continuous document processing but be wasteful for a task executed twice a day.

The goal is not to prove that local inference is always superior to cloud inference.

It is to understand when local execution offers the right balance of privacy, cost, quality, performance, and operational control.

Your call to action: Pick one workload you currently run through a hosted service, reproduce a small representative version on your own hardware, and measure the results. Start with the smallest suitable model, validate its outputs, benchmark both cold and steady-state performance, and only scale up when the evidence justifies it.

The real benefit of learning local AI isn’t merely that you can run a model on your laptop. It’s that you gain the knowledge to decide where inference should run, how it should be deployed, and what trade-offs are worth making.