Serving many models on one GPU: Triton, model repositories, and hot-swapping

LEARN · MULTI-MODEL INFERENCE & SERVING

Why one model per GPU is often the wrong abstraction

A GPU is not a Kubernetes pod. Treating every model as if it deserves an entire accelerator can leave expensive hardware mostly idle, especially when you serve small classifiers, regressors, ranking models, feature transforms, computer-vision models with sporadic traffic, or tenant-specific model variants.

Triton Inference Server is designed for the opposite pattern: one server process can load multiple models, schedule them independently, and execute different models concurrently on the same GPU. Triton can also create multiple execution instances of an individual model when that improves throughput.

As of August 2026, the current Triton Inference Server release is 2.71.0, corresponding to NVIDIA’s 26.07 container release. The examples below therefore use nvcr.io/nvidia/tritonserver:26.07-py3 instead of an older image copied from a years-old tutorial.

The important architectural distinction is this:

A model repository tells Triton which models exist.
A model configuration tells Triton how each model should be scheduled.
A model instance is an executable copy of a model.
Dynamic batching combines compatible requests before execution.
Model control mode determines when models enter or leave the running server.
The GPU is a shared execution resource underneath all of them.

That means the unit you deploy does not have to be “one GPU, one model.” It can be “one GPU, a catalog of models, each loaded only when needed.”

We will build exactly that.

What we are going to run

The lab uses three tiny ONNX models:

scaler_blue: computes OUTPUT0 = INPUT0 + 1.
scaler_green: computes OUTPUT0 = INPUT0 + 2.
multiplier: computes OUTPUT0 = INPUT0 × 2.

scaler_blue and scaler_green represent two releases of the same logical service. Keeping different names gives us a production-friendly blue/green deployment mechanism.

multiplier represents an unrelated second workload sharing the same GPU.

The repository will eventually look like this:

model_repository/
├── multiplier/
│   ├── 1/
│   │   └── model.onnx
│   └── config.pbtxt
├── scaler_blue/
│   ├── 1/
│   │   └── model.onnx
│   └── config.pbtxt
└── scaler_green/
    ├── 1/
    │   └── model.onnx
    └── config.pbtxt

This follows Triton’s current model-repository convention: every model gets its own directory, each model version lives in a numerically named subdirectory, and an ONNX model defaults to the filename model.onnx. Triton can serve repositories from local storage as well as supported object-storage backends such as S3, Google Cloud Storage, and Azure Storage.

Prerequisites

You need Docker, the NVIDIA Container Toolkit, and a supported NVIDIA GPU/driver combination.

First verify that the host can see the GPU:

nvidia-smi

Then verify Docker GPU passthrough:

docker run --rm --gpus=all nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

The exact CUDA utility image is not important to the architecture. What matters is that a container can enumerate the GPU before you start debugging Triton itself.

Create a working directory and Python environment:

mkdir triton-multimodel
cd triton-multimodel
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install onnx

Generate three runnable ONNX models

We do not need a training dataset for this serving exercise. Tiny deterministic graphs are better because they make it immediately obvious whether requests are reaching the intended model.

Create generate_models.py:

from pathlib import Path

import onnx
from onnx import TensorProto, helper


ROOT = Path("model_repository")


CONFIG = """backend: "onnxruntime"
max_batch_size: 32

input [
  {
    name: "INPUT0"
    data_type: TYPE_FP32
    dims: [ 4 ]
  }
]

output [
  {
    name: "OUTPUT0"
    data_type: TYPE_FP32
    dims: [ 4 ]
  }
]

dynamic_batching {
  max_queue_delay_microseconds: 1000
}

instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]
"""


def build_model(name: str, operator: str, value: float) -> None:
    version_dir = ROOT / name / "1"
    version_dir.mkdir(parents=True, exist_ok=True)

    input_info = helper.make_tensor_value_info(
        "INPUT0",
        TensorProto.FLOAT,
        ["batch", 4],
    )
    output_info = helper.make_tensor_value_info(
        "OUTPUT0",
        TensorProto.FLOAT,
        ["batch", 4],
    )

    constant = helper.make_tensor(
        name="VALUE",
        data_type=TensorProto.FLOAT,
        dims=[1],
        vals=[value],
    )

    node = helper.make_node(
        operator,
        inputs=["INPUT0", "VALUE"],
        outputs=["OUTPUT0"],
    )

    graph = helper.make_graph(
        nodes=[node],
        name=name,
        inputs=[input_info],
        outputs=[output_info],
        initializer=[constant],
    )

    model = helper.make_model(
        graph,
        producer_name="triton-multimodel-demo",
        opset_imports=[helper.make_operatorsetid("", 21)],
    )

    onnx.checker.check_model(model)
    onnx.save(model, version_dir / "model.onnx")

    config_path = ROOT / name / "config.pbtxt"
    config_path.write_text(CONFIG, encoding="utf-8")


build_model("scaler_blue", "Add", 1.0)
build_model("scaler_green", "Add", 2.0)
build_model("multiplier", "Mul", 2.0)

print(f"Created model repository at {ROOT.resolve()}")

Generate the repository:

python generate_models.py

Check it:

find model_repository -type f | sort

You should see:

model_repository/multiplier/1/model.onnx
model_repository/multiplier/config.pbtxt
model_repository/scaler_blue/1/model.onnx
model_repository/scaler_blue/config.pbtxt
model_repository/scaler_green/1/model.onnx
model_repository/scaler_green/config.pbtxt

Understanding `config.pbtxt`

The ONNX file contains the computation. config.pbtxt contains Triton’s serving policy.

A minimal Triton configuration describes the backend, maximum batch size, and input/output tensors. Input and output names must correspond to the tensors exposed by the model.

Our important lines are:

backend: "onnxruntime"
max_batch_size: 32

backend selects Triton’s ONNX Runtime backend.

max_batch_size: 32 says Triton may treat the first dimension as a batch dimension and construct batches of up to 32 samples. The model itself declares a shape equivalent to [batch, 4], while the Triton configuration declares only the non-batch dimensions:

dims: [ 4 ]

Next comes:

dynamic_batching {
  max_queue_delay_microseconds: 1000
}

Dynamic batching lets Triton combine compatible inference requests into a larger execution batch. It is configured independently for each model. The optional queue delay lets the scheduler briefly wait for additional requests instead of dispatching every request immediately.

That does not mean one millisecond is universally optimal. It means we have explicitly chosen a one-millisecond batching budget for this demonstration. An interactive API might use a smaller value or no deliberate delay; a throughput-oriented service might tolerate more.

Finally:

instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

This creates one execution instance on GPU 0.

Do not confuse “one instance” with “one model on the GPU.” Every loaded model can have its own instance group. Triton’s scheduler can execute different loaded models concurrently on the same GPU.

Start Triton without loading any models

For dynamic model fleets, the most useful control mode is usually explicit.

Start Triton:

docker run --rm \
  --gpus=all \
  --name triton-multimodel \
  -p 8000:8000 \
  -p 8001:8001 \
  -p 8002:8002 \
  -v "$PWD/model_repository:/models" \
  nvcr.io/nvidia/tritonserver:26.07-py3 \
  tritonserver \
  --model-repository=/models \
  --model-control-mode=explicit

Triton exposes HTTP on port 8000, gRPC on 8001, and its metrics service on port 8002 in the standard server configuration.

More importantly, explicit mode changes startup behavior.

With no --load-model option, Triton starts without loading models. You then decide which models become resident through the repository-control API. In explicit mode, subsequent load and unload operations are initiated through the model-control protocol.

That is precisely what we want when 100 models exist in storage but perhaps only 12 need GPU memory right now.

Inspect the model catalog

Open another terminal in the project directory.

Ask Triton what exists in the repository:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{}' \
  http://localhost:8000/v2/repository/index \
  | python -m json.tool

The repository extension exposes an index operation alongside model load and unload operations. The index can include models that exist in the repository even when they are not currently ready for inference.

Conceptually, the distinction is:

Repository:
  scaler_blue
  scaler_green
  multiplier

Currently resident on GPU:
  nothing

That separation is one of the foundations of multi-model serving.

Load two models onto one GPU

Load the blue scaler:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_blue/load

Load the unrelated multiplier:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/multiplier/load

Check only ready models:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"ready":true}' \
  http://localhost:8000/v2/repository/index \
  | python -m json.tool

We now have two independent model instances sharing GPU 0.

The repository API’s current HTTP endpoints are:

POST /v2/repository/index
POST /v2/repository/models/<model-name>/load
POST /v2/repository/models/<model-name>/unload

A successful load means Triton has completed the requested load operation; the same repository extension is also available over gRPC.

Send inference requests to both models

Call scaler_blue:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "name": "INPUT0",
        "shape": [1, 4],
        "datatype": "FP32",
        "data": [1.0, 2.0, 3.0, 4.0]
      }
    ]
  }' \
  http://localhost:8000/v2/models/scaler_blue/infer \
  | python -m json.tool

Its output data should represent:

[2.0, 3.0, 4.0, 5.0]

Now send the same input to multiplier:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "name": "INPUT0",
        "shape": [1, 4],
        "datatype": "FP32",
        "data": [1.0, 2.0, 3.0, 4.0]
      }
    ]
  }' \
  http://localhost:8000/v2/models/multiplier/infer \
  | python -m json.tool

This time the output represents:

[2.0, 4.0, 6.0, 8.0]

There is no second Triton container, second HTTP stack, or second GPU allocation. Both models are independently addressable inside one inference server.

Under simultaneous traffic, Triton’s architecture allows requests targeting different models to be scheduled onto the same GPU concurrently.

What “hot-swapping” actually means in Triton

This is where architecture matters.

There are three current model-control modes:

none
explicit
poll

The default, none, loads the repository at startup and ignores subsequent repository changes.

poll periodically notices repository changes and can reload models when files change.

explicit gives your deployment controller direct load/unload operations.

It is tempting to use poll as a magical production hot-reload mechanism: copy a new model into a directory and wait.

Do not build your production rollout process around that assumption.

Current Triton documentation explicitly warns that repository polling is not recommended for production, because a polling cycle can observe a repository while files are only partially updated. Although a successful poll-mode reload can replace an existing model without losing availability, synchronization between your filesystem update and Triton’s poll does not exist.

Explicit control gives you a much cleaner deployment state machine.

Same-name reload versus blue/green replacement

Suppose scaler_blue is version A and you want to replace its files with version B.

In explicit mode, current Triton guidance says an already loaded model should be explicitly unloaded before the updated model is loaded.

The simple sequence is therefore:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_blue/unload

Then update its repository files and load it again:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_blue/load

That is a hot operational update in the sense that the Triton server itself stays running and unrelated models remain available.

But it is not a zero-downtime guarantee for requests targeting scaler_blue during the gap.

For high-availability production deployments, use blue/green model identities.

A production-friendly blue/green swap

We already have scaler_green in the repository, computing the new behavior.

Load it while blue remains online:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_green/load

Check readiness:

curl -f \
  http://localhost:8000/v2/models/scaler_green/ready

Smoke-test the new release:

curl -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "name": "INPUT0",
        "shape": [1, 4],
        "datatype": "FP32",
        "data": [1.0, 2.0, 3.0, 4.0]
      }
    ]
  }' \
  http://localhost:8000/v2/models/scaler_green/infer \
  | python -m json.tool

The new result should be:

[3.0, 4.0, 5.0, 6.0]

Your application gateway, inference router, service-discovery layer, or deployment controller can now change the logical mapping:

logical service "scaler" -> scaler_blue

to:

logical service "scaler" -> scaler_green

Only after traffic has drained from the previous deployment do you unload blue:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_blue/unload

The GPU residency sequence becomes:

Before deployment:
  scaler_blue
  multiplier

During deployment:
  scaler_blue
  scaler_green
  multiplier

After cutover:
  scaler_green
  multiplier

The overlap consumes extra GPU memory temporarily, but it avoids coupling model replacement to a same-name unload/load gap.

That is a much more useful definition of hot-swapping in a production system: load the replacement, verify it, route traffic, then retire the old model.

Scaling beyond two models

Nothing in this architecture says model A and model B must consume half the GPU each.

A small model might execute for a fraction of a millisecond and consume relatively little device memory. Another might occupy most of the GPU and dominate the scheduler.

Triton’s model-instance configuration lets you control this per model.

For example:

instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

creates two executable instances of that model on GPU 0.

Multiple instances can improve concurrency for models that cannot saturate the GPU with one execution stream. Triton also supports multiple different models executing concurrently; instance groups simply add another dimension of concurrency.

Do not automatically set count: 4 on every model.

Every additional instance can consume memory, backend resources, execution contexts, and scheduler capacity. The correct number is a benchmarking result, not a stylistic preference.

Dynamic batching and model concurrency solve different problems

These two features are often conflated.

Dynamic batching asks:

Can several compatible requests for the same model become one larger inference execution?

Multiple instances ask:

Can more than one inference execution of this model run concurrently?

Multi-model execution asks:

Can different models execute concurrently on the shared device?

You can combine all three.

For example, imagine four recommendation models receiving small requests:

model A queue -> batch -> instance A1
model B queue -> batch -> instance B1
model C queue -> batch -> instance C1
model D queue -> batch -> instance D1

                           shared GPU

The scheduler can batch requests independently for each model while multiple model executions compete for GPU execution resources.

That is dramatically different from running four containers that each believe they exclusively own the accelerator.

Protecting the GPU from overcommit

Concurrency is useful until it becomes uncontrolled contention.

Triton includes a rate limiter that works across loaded models. It can postpone model-instance execution when running every eligible model simultaneously would overload the system. Its model-instance configuration also supports resources and priorities for cross-model scheduling.

This matters when models have very different behavior.

For example:

Model A allocates significant temporary GPU memory.
Model B has strict latency requirements.
Model C is a low-priority batch workload.
Model D has several execution instances.
Model E occasionally produces a traffic burst.

Without a resource policy, “everything can run concurrently” can turn into “everything becomes slow concurrently.”

Rate limiting lets you model scarce execution resources instead.

At a higher level, you should also consider separate GPU pools when workloads have fundamentally incompatible service-level objectives. Sharing works best when you understand the interference profile rather than merely discovering that all models technically fit in memory.

The cherry on the cake: one GPU can plausibly hold hundreds of tiny models

A surprisingly common mental limit is something like “maybe four models per GPU.”

Triton does not define that sort of small fixed model-count ceiling. The current execution architecture allows multiple different models and instances to share a GPU; your practical limits come from things such as model memory, backend/session overhead, workspace requirements, throughput, latency interference, and available compute.

Consider a purely illustrative 24 GiB GPU.

Suppose measurements on your actual workload show:

Triton, backend state, and reserved headroom consume 1.5 GiB.
Each small resident model adds approximately 80 MiB.

A memory-only capacity estimate is easy:

gpu_gib = 24.0
baseline_gib = 1.5
per_model_mib = 80.0

available_mib = (gpu_gib - baseline_gib) * 1024
memory_only_limit = int(available_mib // per_model_mib)

print(memory_only_limit)

The output is:

That does not mean you should promise 288 simultaneously busy models on every 24 GiB GPU.

It means something more interesting: memory residency alone can put the theoretical count in the hundreds rather than the single digits when individual models are small.

Actual usable concurrency may be 30, 80, 150, or another number entirely. If all 288 suddenly receive compute-heavy requests at once, latency could become unacceptable even though they fit in memory.

Measure actual incremental residency instead of guessing.

For example, inspect GPU memory before loading models:

nvidia-smi \
  --query-gpu=memory.total,memory.used,memory.free \
  --format=csv

Load one representative model:

curl -f \
  -X POST \
  http://localhost:8000/v2/repository/models/scaler_blue/load

Inspect memory again:

nvidia-smi \
  --query-gpu=memory.total,memory.used,memory.free \
  --format=csv

Repeat with representative models and execution-instance counts.

The important production number is not “How many directories can Triton see?”

It is:

How many resident model instances can this GPU sustain while meeting the latency and throughput objectives of the whole fleet?

That is a capacity-planning question, not a configuration-file question.

Observe the server while models move in and out

Triton exposes metrics separately from the inference HTTP and gRPC endpoints.

You can inspect the Prometheus-format endpoint directly:

curl -s http://localhost:8002/metrics | head -n 40

During development, combine several views:

Triton metrics for request and execution behavior.
Repository index for model state.
Model readiness endpoints during deployment.
nvidia-smi for coarse GPU memory and utilization checks.
A real load generator for latency and throughput.
Production traces around your routing layer.

Do not validate GPU sharing with one sequential curl command. Concurrency behavior becomes visible only under concurrent traffic.

A good benchmark should represent the mixture you expect in production.

If your actual traffic is:

50% model A
20% model B
15% model C
10% model D
5% model E

benchmarking each model independently at maximum throughput does not tell you whether the combined deployment works.

The fleet is the workload.

An important security consequence of hot-swapping

Dynamic model management is an operational capability and a security boundary.

NVIDIA’s current secure-deployment guidance explicitly warns that allowing dynamic updates to model repositories can lead to arbitrary code execution if repository access is compromised. It recommends disabling dynamic updates unless they are required and carefully restricting access when they are enabled.

That warning makes sense once you remember that Triton can support executable model backends and model artifacts rather than treating the repository as passive data.

Do not expose model-control endpoints directly to the public internet.

A production design should normally ensure that:

Only a trusted deployment controller can alter model artifacts.
Only authorized infrastructure can invoke load/unload operations.
Repository writes are authenticated and audited.
Artifacts are validated before becoming loadable.
Model names and versions come from a controlled release process.
The inference-facing identity does not automatically receive deployment privileges.
Network policy isolates management operations from ordinary inference traffic.

The same API that makes hot-swapping convenient is powerful enough to deserve deployment-grade access control.

Why explicit control usually beats repository polling

Repository polling appears simpler because no controller needs to call an API.

But simplicity disappears when you ask production questions:

Has the entire new model finished uploading?
Which process decides that it is safe to activate?
How do we run a smoke test before traffic arrives?
How do we roll back?
How do we know which release is currently active?
How do we avoid a poll seeing half of an update?

Current Triton documentation specifically warns that poll mode has no synchronization with repository modifications and is not recommended for production.

With explicit control, your deployment process can instead be deterministic:

1. Publish immutable artifact.
2. Verify artifact.
3. Make new model visible in repository.
4. Request load.
5. Wait for successful load/readiness.
6. Smoke-test inference.
7. Shift traffic.
8. Observe.
9. Drain old model.
10. Request unload.
11. Delete old artifact later.

That workflow also maps cleanly onto CI/CD systems and Kubernetes operators.

Do not mutate files while Triton is loading them

There is another subtle repository rule worth knowing.

Triton’s model-management documentation distinguishes model states when discussing repository changes. While a model is actively loading or unloading, files inside its model directory must not be added, removed, or modified. When a model has been completely unloaded, its model directory can safely be changed.

That is another reason immutable release directories are easier to reason about.

Prefer:

scaler_blue
scaler_green

or immutable versioned deployment identities over a script that continuously overwrites:

production_model/1/model.onnx

while simultaneously telling the inference server to reload it.

Deployment infrastructure becomes far easier to debug when artifacts have immutable identities and activation is a separate operation.

A practical multi-model deployment policy

For a large catalog, classify models by both traffic and residency cost.

A useful policy might look like:

Tier 1:
  High traffic
  Always loaded

Tier 2:
  Moderate traffic
  Loaded during active periods

Tier 3:
  Rarely used
  Load on demand, unload after inactivity

Tier 4:
  Batch-only
  CPU or dedicated batch GPU pool

That opens the door to an inference controller that treats GPU memory as a cache.

A simplified controller loop can be conceptualized as:

def reconcile(model, desired, actual):
    if desired == "loaded" and actual != "loaded":
        load_model(model)

    if desired == "unloaded" and actual == "loaded":
        unload_model(model)

Real infrastructure will also need locking, failure handling, health checks, rollout state, routing changes, backoff, capacity checks, and observability.

But the important insight is that model residency becomes dynamic state.

Your object store or model repository may contain thousands of artifacts. Your GPU only needs the active working set.

When GPU sharing works especially well

Multi-model serving is particularly attractive when you have:

Many relatively small models.
Tenant-specific models with uneven request rates.
Geographically partitioned models.
Ranking or recommendation models with bursty traffic.
Several computer-vision models that are not simultaneously saturated.
Multiple stages whose combined GPU utilization fits comfortably on one device.
A long tail of rarely used models that can be loaded and unloaded.
Several versions temporarily coexisting during deployments.

It becomes less attractive when one model already saturates the device, when models require almost all GPU memory, or when strict tail-latency requirements make cross-workload interference unacceptable.

“Can they share?” and “should they share?” are different questions.

A production checklist

Before consolidating many production models onto one GPU, verify each of these.

Repository design

Give every model an immutable artifact.
Use numeric Triton version directories correctly.
Avoid editing a model while it is loading or unloading.
Separate artifact publication from model activation.

Scheduling

Benchmark instance_group counts rather than guessing.
Enable dynamic batching only where request semantics allow it.
Tune queue delay against actual latency objectives.
Test realistic multi-model traffic mixtures.
Consider rate limiting when workloads compete aggressively.

Deployment

Prefer explicit model control for orchestrated dynamic fleets.
Use blue/green identities when you need continuous availability.
Verify readiness before shifting traffic.
Keep the previous release resident long enough for practical rollback.
Unload inactive models to recover capacity.

Security

Restrict repository write access.
Restrict load/unload APIs.
Treat model artifacts as executable supply-chain inputs.
Audit deployment actions.
Keep management endpoints behind trusted infrastructure.

Capacity

Measure actual per-model GPU-memory deltas.
Include execution-instance overhead.
Preserve memory headroom.
Measure p95 and p99 latency under mixed traffic.
Plan for temporary blue/green overlap during releases.

Where to go from here

At this point, the single-model mental model should be gone.

Triton can act as a model-serving runtime for an entire GPU, not merely as a wrapper around one neural network. A repository can contain many models; explicit model control determines which ones are resident; dynamic batching improves per-model request packing; instance groups control intra-model concurrency; and the scheduler lets multiple models share the accelerator.

The architecture scales from the three tiny ONNX graphs in this tutorial to much more interesting fleets:

GPU 0
├── ranking_eu_v17
├── ranking_us_v32
├── fraud_rules_v8
├── image_embedding_v12
├── category_classifier_v5
├── tenant_104_model
├── tenant_283_model
└── next release being warmed up

The next exercise is to make this quantitative: replace the toy graphs with two or three real models, drive them concurrently, measure GPU-memory residency and tail latency, then vary dynamic batching and instance_group counts.

Build that benchmark before buying another GPU. You may discover that the accelerator you thought could serve one model comfortably can serve an entire model fleet.

Why one model per GPU is often the wrong abstraction

What we are going to run

Prerequisites

Generate three runnable ONNX models

Understanding `config.pbtxt`

Start Triton without loading any models

Inspect the model catalog

Load two models onto one GPU

Send inference requests to both models

What “hot-swapping” actually means in Triton

Same-name reload versus blue/green replacement

A production-friendly blue/green swap

Scaling beyond two models

Dynamic batching and model concurrency solve different problems

Protecting the GPU from overcommit

The cherry on the cake: one GPU can plausibly hold hundreds of tiny models

Observe the server while models move in and out

An important security consequence of hot-swapping

Why explicit control usually beats repository polling

Do not mutate files while Triton is loading them

A practical multi-model deployment policy

When GPU sharing works especially well

A production checklist

Where to go from here

Vulnerability Watch No38

Tech Layoffs Watch No12

ELT and dbt: transforming data where it lives, with tests and lineage

Serving many models on one GPU: Triton, model repositories, and hot-swapping

Why one model per GPU is often the wrong abstraction

What we are going to run

Prerequisites

Generate three runnable ONNX models

Understanding config.pbtxt

Start Triton without loading any models

Inspect the model catalog

Load two models onto one GPU

Send inference requests to both models

What “hot-swapping” actually means in Triton

Same-name reload versus blue/green replacement

A production-friendly blue/green swap

Scaling beyond two models

Dynamic batching and model concurrency solve different problems

Protecting the GPU from overcommit

The cherry on the cake: one GPU can plausibly hold hundreds of tiny models

Observe the server while models move in and out

An important security consequence of hot-swapping

Why explicit control usually beats repository polling

Do not mutate files while Triton is loading them

A practical multi-model deployment policy

When GPU sharing works especially well

A production checklist

Where to go from here

Vulnerability Watch No38

Tech Layoffs Watch No12

ELT and dbt: transforming data where it lives, with tests and lineage

Understanding `config.pbtxt`