GPU right-sizing and autoscaling: serving models without burning money

A GPU serving a machine learning model can be expensive even when it is doing almost nothing.

The problem is rarely just the price of the accelerator. It is the combination of oversized hardware, inefficient batching, unpredictable traffic, slow startup times, and infrastructure that continues running after demand disappears.

A model serving 100 requests per second during peak hours might receive only five requests per second overnight. If the deployment keeps enough GPUs online for peak traffic, the organization pays for unused capacity for most of the day.

Autoscaling promises to fix this, but applying ordinary CPU-based scaling rules to GPU inference is often insufficient. GPU memory, model-loading time, batching efficiency, token generation, and queueing behavior create constraints that traditional web services do not encounter.

The goal is not maximum GPU utilization at any cost. It is to minimize infrastructure spending while satisfying measurable latency, throughput, availability, and reliability requirements.

In this guide, we will:

• Understand the economics of GPU inference and identify avoidable spending.

• Choose appropriately sized accelerators based on model memory, throughput, and latency.

• Design scaling policies that react to useful workload signals rather than misleading utilization numbers.

• Build a runnable Python simulation comparing fixed provisioning with load-aware scaling.

• Translate the simulation into practical Kubernetes scaling and infrastructure recommendations.

• Calculate savings, identify operational risks, and build a production rollout plan.

1. Why idle GPU capacity becomes an expensive mistake

A common trap: the GPU that comes back after you terminate it

A familiar story in cloud cost post-mortems goes like this. A team launches a GPU Spot instance for a benchmark and terminates it when the experiment finishes. But the instance was created through a persistent Spot request. Terminating the instance does not cancel that request, so EC2 launches a replacement. Nobody uses it, and it bills for days until someone notices a GPU-sized line item on the invoice.

The underlying mechanism is documented by AWS: a persistent Spot request can launch a new instance after the previous instance is terminated. Canceling the request and terminating the running instance are separate lifecycle operations.

The lesson is broader than Spot Instances.

An idle GPU is not automatically a free GPU.

An accelerator can remain billable while:

• A deployment has zero incoming requests.

• A model server is running but not performing inference.

• A Kubernetes pod has terminated, but its GPU node remains allocated.

• A persistent infrastructure request recreates compute after termination.

• A reserved or committed allocation remains financially obligated even when unused.

Autoscaling must therefore control the entire resource lifecycle, not merely the number of application replicas.

The difference between allocated, active, and productive capacity

Three quantities are worth measuring separately.

Allocated capacity is the hardware provisioned for the workload. It determines much of the infrastructure bill.

Active capacity is the fraction of that hardware performing operations during a measurement interval.

Productive capacity is the work that successfully contributes to completed inference requests meeting the required service objectives.

These are not interchangeable.

A GPU can report high activity while processing inefficient batches, repeatedly recomputing redundant prefixes, or spending resources on requests that clients have already canceled.

Conversely, moderate activity might be exactly what a latency-sensitive service needs to preserve headroom for bursts.

The useful optimization target is:

Cost per successful request meeting the service-level objective (SLO).

For generative models, cost per successfully delivered output token is another useful metric, provided that input-token processing, request length, caching, and quality requirements are accounted for.

Both metrics connect infrastructure expenditure to actual application value.

2. GPU right-sizing: choose the smallest hardware that satisfies the workload

Right-sizing means selecting the combination of GPU type, GPU count, memory capacity, runtime configuration, and concurrency settings that satisfies the application’s requirements at the lowest sustainable cost.

It is not simply choosing the cheapest instance.

A less expensive GPU that processes half as many requests at the same latency can increase total spending if you need three times as many instances.

Likewise, using an expensive accelerator for a small embedding model may provide impressive benchmark throughput without improving the application’s actual response time.

Understand the current GPU instance landscape

As of October 2026, cloud providers offer several materially different GPU deployment shapes.

These configurations are documented in the current AWS G6 specifications and Google Cloud GPU machine reference.

Pricing must be interpreted carefully because cloud providers package accelerators differently.

For example:

• An AWS g6.xlarge running Linux in us-east-1 was listed at $0.8048 per hour in September 2026. That is an instance price covering its GPU, CPU, and included memory allocation, not merely an accelerator surcharge. See the current instance pricing reference.

• Google Cloud lists the NVIDIA T4 GPU component at $0.35 per GPU-hour in its US central pricing selection, before additional VM and other resource charges. Its official GPU pricing page separates GPU component pricing from machine-type charges for applicable families.

• Google Cloud G2, A2, A3, and A4 shapes are priced as accelerator-optimized VM configurations, so their bundled instance prices must be used for fair comparison.

The exact price depends on region, operating system, purchasing commitment, discounts, and deployment model. Availability and quotas matter too.

The correct comparison is not simply dollars per GPU-hour.

It is dollars per unit of SLO-compliant inference work, including the supporting infrastructure.

Calculate the baseline bill

Consider ten AWS g6.xlarge instances operating continuously for a 730-hour billing month.

The compute cost is:

Cost = GPU-instance count × hours × instance price

Cost = 10 × 730 × $0.8048 = $5,875.04 per month

This excludes incremental storage, networking, load balancing, monitoring, managed Kubernetes fees, and other charges.

Now assume that a load-aware deployment averages four instances instead of ten.

Cost = 4 × 730 × $0.8048 = $2,350.02 per month

That produces approximately $3,525 in monthly compute savings, or 60%, provided the infrastructure really releases the six unnecessary instances and there are no overlapping committed charges.

The same arithmetic becomes more significant with expensive multi-GPU instances.

Model memory is often the first constraint

Before benchmarking throughput, determine whether the model fits into GPU memory with sufficient operating headroom.

GPU memory consumption includes:

• Model weights.

• Quantization scales and metadata.

• Runtime workspaces and temporary tensors.

• Attention key-value caches.

• Memory consumed by concurrent requests.

• Framework and allocator overhead.

Consider an illustrative model with seven billion parameters.

With two bytes per weight, its uncompressed BF16 weight storage is approximately 14 GB in decimal units. That does not include runtime overhead or KV cache.

Four-bit weights reduce the theoretical raw weight footprint to approximately 3.5 GB, but actual quantized storage is larger because of metadata, alignment, and implementation details.

The apparent space savings can enable smaller GPUs, larger batches, or longer contexts, but quantization can also change throughput and output quality. Those effects must be measured.

Why the KV cache can change your hardware decision

For a transformer using a conventional per-layer key-value cache, an approximate memory model is:

KV bytes = 2 × layers × KV heads × head dimension × bytes per element × cached tokens

The factor of two accounts for keys and values.

Suppose an illustrative architecture has:

• 32 transformer layers.

• 8 KV heads per layer.

• A head dimension of 128.

• Two bytes per cached element.

• An 8,192-token sequence.

This produces approximately 128 KiB of KV cache per token, or 1 GiB per 8,192-token sequence.

Sixteen such concurrent sequences could require approximately 16 GiB of KV storage alone.

The exact result depends on architecture, cache precision, paging, prefix sharing, and attention implementation. Nevertheless, it demonstrates why a model that fits comfortably at concurrency one can run out of memory under realistic serving conditions.

For large language models, selecting hardware using model weight size alone is a common and expensive sizing mistake.

Benchmark throughput at the intended latency

Suppose performance testing determines that one GPU can process 60 representative requests per minute at the configured batching and request-length distribution.

The service expects peak demand of 240 requests per minute.

If we deliberately target 70% of benchmark throughput to preserve operational headroom:

Safe capacity per GPU = 60 × 0.70 = 42 requests/minute

Required GPUs = ceiling(240 / 42) = 6 GPUs

The headroom is an engineering choice, not a universal rule. A 70% target is appropriate only if measurements show that latency and stability remain acceptable near that operating point.

In production, repeat the experiment with varied:

• Input and output token lengths.

• Batch sizes and concurrency.

• Cache-hit rates.

• Streaming versus non-streaming requests.

• Multiple model variants or tenant mixes.

• Realistic burst patterns.

A GPU’s maximum throughput is rarely its maximum throughput at acceptable latency.

3. Choose scaling signals that reflect actual demand

Autoscaling quality depends heavily on what the controller observes.

CPU-based autoscaling works reasonably well for many stateless applications because CPU pressure often correlates with workload pressure.

GPU inference is more complicated.

Why GPU utilization alone is not enough

NVIDIA’s DCGM Exporter exposes metrics including GPU activity, framebuffer memory usage, power, and hardware health.

These measurements are useful for diagnosis and capacity planning. They do not fully describe application demand.

For example:

• GPU activity can be high because large batches are running efficiently, while request latency remains healthy.

• GPU activity can be low while a model server is bottlenecked on CPU-side tokenization, scheduling, or network I/O.

• Memory allocation can remain high because model weights and caches stay resident even when request volume falls.

• A saturated GPU can have growing queues even if its utilization metric has already reached a plateau.

GPU utilization should therefore be one input to the controller, not the only input.

Metrics worth collecting

For online inference, queue age and latency are especially valuable because they connect scaling decisions to user experience.

For generative serving, separating prefill demand from decode demand can improve scaling further. A request containing 10,000 input tokens and producing 20 output tokens has a very different resource profile from one containing 100 input tokens and producing 2,000 output tokens.

Use a capacity-based scaling equation

A practical starting point is:

Desired replicas = ceiling(expected requests/minute / safe requests/minute per replica)

Then clamp the result between a minimum and maximum number of replicas.

For example:

• Expected demand: 300 requests/minute.

• Benchmark capacity: 60 requests/minute per GPU.

• Target utilization: 70%.

• Safe capacity: 42 requests/minute per GPU.

Desired replicas = ceiling(300 / 42) = 8

The controller should also account for waiting work. A simple planning rule is:

Effective demand = predicted arrivals + queue backlog / desired drain time

The units matter: if arrivals are measured per minute, queue backlog must be divided by a drain interval measured in minutes.

This allows a controller to allocate additional capacity when the queue is growing, rather than merely matching the latest arrival rate.

For real systems, use a rate predictor, smoothing, and configurable safeguards. A single unexpectedly large sample should not repeatedly trigger expensive scale-up and scale-down cycles.

Why scaling must be asymmetric

The costs of scaling up and scaling down are not equal.

Under-provisioning can immediately cause:

• Higher response latency.

• Request timeouts.

• Rejected traffic.

• Failed user interactions.

• Queue accumulation.

Over-provisioning primarily creates additional expense.

For latency-sensitive applications, a common starting policy is therefore:

• Scale up aggressively when demand exceeds safe capacity.

• Scale down conservatively after demand remains low.

• Maintain a minimum warm pool where cold-start latency is unacceptable.

• Use known traffic events to provision capacity ahead of time.

• Preserve additional capacity during hardware failures or availability-zone disruptions.

These principles help prevent oscillation without treating cost efficiency as more important than service reliability.

4. Hands-on: build a runnable GPU autoscaling simulation

We will now build a discrete-time simulation of a GPU-backed inference service.

Instead of downloading a benchmark dataset, the simulation generates synthetic e-commerce traffic. This makes the experiment reproducible, independent of external services, and safe to run without access to GPUs.

The simulation models an online product-ranking service that receives requests throughout the day.

We will compare three infrastructure policies:

1. Fixed provisioning: Keep ten GPUs running continuously.

2. Reactive scaling: Adjust capacity based on arriving requests and accumulated backlog.

3. Predictive prewarming: Use reactive scaling but prepare additional GPUs before a known promotional traffic spike.

Define the assumptions

The compute price is the illustrative AWS g6.xlarge on-demand price discussed earlier.

The four-minute startup delay is an assumed modeling parameter, not a guaranteed cloud-provider startup time. Real startup includes VM provisioning, scheduling, image pulls, model initialization, and readiness checks.

We also make several simplifying assumptions:

• Requests have equal average processing cost.

• A fully ready GPU can serve a fixed number of requests each minute.

• New GPUs become billable as soon as provisioning begins.

• A request can be served in the minute it arrives if capacity exists.

• There are no infrastructure failures, interruptions, or request retries.

• The simulator measures queue wait in whole-minute buckets rather than full end-to-end inference latency.

These simplifications make the economic trade-offs easier to see. They do not replace production load testing.

Step 1: Create the project

The simulation requires Python 3.10 or newer and uses only the standard library.

Create a new directory and optionally isolate the environment:

mkdir gpu-scaling-lab
cd gpu-scaling-lab

python -m venv .venv
source .venv/bin/activate

On Windows, activate the environment using the appropriate activation script for your shell.

Create a file named gpu_scaling_sim.py and add the complete implementation below.

Step 2: Implement the simulator

"""Minute-resolution GPU serving capacity simulation, standard library only."""

from collections import Counter, deque
from math import ceil
from random import Random

MINUTES = 24 * 60
GPU_REQUESTS_PER_MIN = 60
HOURLY_PRICE = 0.8048  # Illustrative AWS g6.xlarge, us-east-1

MAX_GPUS = 10
MIN_GPUS = 2
TARGET_UTILIZATION = 0.70
STARTUP_MINUTES = 4
SCALE_DOWN_COOLDOWN = 15


def make_traffic():
    rng = Random(42)
    traffic = []

    for minute in range(MINUTES):
        hour = minute // 60

        if hour < 6:
            baseline = 25
        elif hour < 8:
            baseline = 55
        elif hour < 12:
            baseline = 115
        elif hour < 17:
            baseline = 85
        elif hour < 21:
            baseline = 145
        else:
            baseline = 55

        # A short evening promotion generates a sudden traffic spike.
        if 17 * 60 + 40 <= minute < 18 * 60:
            baseline += 280

        traffic.append(max(0, baseline + rng.randint(-12, 12)))

    return traffic


def percentile(counts, q):
    n = sum(counts.values())
    cutoff = ceil(n * q)
    seen = 0

    for wait in sorted(counts):
        seen += counts[wait]

        if seen >= cutoff:
            return wait

    return 0


def simulate(traffic, mode):
    active = MAX_GPUS if mode == "fixed" else MIN_GPUS

    pending = []  # Entries: (ready_minute, GPU_count)
    queue = deque()  # FIFO entries: (arrival_minute, request_count)
    wait_histogram = Counter()

    billed_gpu_minutes = 0
    useful_gpu_minutes = 0.0
    max_queue = 0
    cooldown = 0

    for minute, incoming in enumerate(traffic):
        # Newly provisioned GPUs become available.
        ready = sum(
            count for when, count in pending if when <= minute
        )
        active += ready

        pending = [
            (when, count)
            for when, count in pending
            if when > minute
        ]

        backlog = sum(count for _, count in queue)

        if mode != "fixed":
            # Arrivals plus capacity needed to drain existing backlog.
            predicted = incoming + backlog / 3

            desired = max(
                MIN_GPUS,
                min(
                    MAX_GPUS,
                    ceil(
                        predicted
                        / (
                            GPU_REQUESTS_PER_MIN
                            * TARGET_UTILIZATION
                        )
                    ),
                ),
            )

            # Prepare capacity before a scheduled promotion.
            if (
                mode == "prewarm"
                and 17 * 60 + 35 <= minute < 18 * 60 + 5
            ):
                desired = MAX_GPUS

            allocated = active + sum(
                count for _, count in pending
            )

            # Scale up immediately, but capacity has a startup delay.
            if desired > allocated:
                pending.append(
                    (minute + STARTUP_MINUTES, desired - allocated)
                )

            # Scale down only after sustained low demand.
            if desired < active:
                cooldown += 1

                if cooldown >= SCALE_DOWN_COOLDOWN:
                    active = desired
                    cooldown = 0
            else:
                cooldown = 0

        # Provisioning GPUs are assumed billable during warm-up.
        billed_gpu_minutes += active + sum(
            count for _, count in pending
        )

        if incoming:
            queue.append((minute, incoming))

        capacity = active * GPU_REQUESTS_PER_MIN
        served = 0

        # Complete oldest requests first.
        while queue and served < capacity:
            created_at, count = queue.popleft()
            completed = min(count, capacity - served)

            wait_histogram[minute - created_at] += completed
            served += completed

            if count > completed:
                queue.appendleft(
                    (created_at, count - completed)
                )

        useful_gpu_minutes += served / GPU_REQUESTS_PER_MIN

        max_queue = max(
            max_queue,
            sum(count for _, count in queue),
        )

    incomplete = sum(count for _, count in queue)
    total = sum(traffic)

    return {
        "cost": billed_gpu_minutes / 60 * HOURLY_PRICE,
        "gpu_hours": billed_gpu_minutes / 60,
        "served": total - incomplete,
        "incomplete": incomplete,
        "max_queue": max_queue,
        "p95_wait_min": percentile(wait_histogram, 0.95),
        "p99_wait_min": percentile(wait_histogram, 0.99),
        "avg_util": useful_gpu_minutes / billed_gpu_minutes,
        "requests": total,
        "fraction_delayed": (
            1 - wait_histogram[0] / (total - incomplete)
        ),
    }


if __name__ == "__main__":
    traffic = make_traffic()

    fixed = simulate(traffic, mode="fixed")
    scaled = simulate(traffic, mode="reactive")
    prewarmed = simulate(traffic, mode="prewarm")

    results = (
        ("Fixed 10 GPUs", fixed),
        ("Reactive autoscaling", scaled),
        ("Autoscaling + prewarm", prewarmed),
    )

    for name, result in results:
        print(name)
        print(f"  GPU-hours: {result['gpu_hours']:.1f}")
        print(f"  Compute cost: ${result['cost']:.2f}")
        print(
            "  Average productive utilization: "
            f"{result['avg_util']:.1%}"
        )
        print(
            "  p95 / p99 queue wait: "
            f"{result['p95_wait_min']} / "
            f"{result['p99_wait_min']} min"
        )
        print(
            "  Requests delayed >=1 minute: "
            f"{result['fraction_delayed']:.1%}"
        )
        print(
            f"  Maximum backlog: {result['max_queue']} requests"
        )
        print(
            f"  Unserved at end of day: {result['incomplete']}"
        )

    for label, value in (
        ("Reactive", scaled),
        ("Prewarm", prewarmed),
    ):
        savings = fixed["cost"] - value["cost"]
        savings_pct = 1 - value["cost"] / fixed["cost"]

        print(
            f"{label} savings: ${savings:.2f} "
            f"({savings_pct:.1%})"
        )

Step 3: Run the experiment

Execute:

python gpu_scaling_sim.py

The fixed random seed ensures the same traffic pattern on every run.

The output is:

Fixed 10 GPUs
  GPU-hours: 240.0
  Compute cost: $193.15
  Average productive utilization: 13.8%
  p95 / p99 queue wait: 0 / 0 min
  Requests delayed >=1 minute: 0.0%
  Maximum backlog: 0 requests
  Unserved at end of day: 0

Reactive autoscaling
  GPU-hours: 70.7
  Compute cost: $56.90
  Average productive utilization: 46.9%
  p95 / p99 queue wait: 0 / 1 min
  Requests delayed >=1 minute: 2.1%
  Maximum backlog: 749 requests
  Unserved at end of day: 0

Autoscaling + prewarm
  GPU-hours: 71.7
  Compute cost: $57.70
  Average productive utilization: 46.2%
  p95 / p99 queue wait: 0 / 0 min
  Requests delayed >=1 minute: 0.0%
  Maximum backlog: 1 requests
  Unserved at end of day: 0

Reactive savings: $136.25 (70.5%)
Prewarm savings: $135.45 (70.1%)

Step 4: Analyze the results

These are results from the synthetic simulation, not measured cloud savings or demonstrated production latency.

Several findings are important.

Finding 1: Fixed provisioning wastes substantial capacity

The fixed deployment spends $193.15 per simulated day and averages only 13.8% productive capacity usage.

This is not necessarily 13.8% reported GPU hardware utilization. It is the ratio of modeled completed work to purchased service capacity.

The deployment serves every request without any minute-bucket queueing, but it achieves that by paying for maximum capacity throughout the day.

Finding 2: Reactive scaling dramatically reduces costs

Reactive scaling reduces simulated spending to $56.90 per day.

It saves 70.5% because the controller adds GPUs during busy periods and releases capacity when demand declines.

However, it temporarily accumulates 749 waiting requests.

During a sudden promotion, the existing GPUs cannot immediately absorb the extra workload. The controller requests additional capacity, but the four-minute startup delay allows the queue to grow.

Even though only 2.1% of completed requests wait into a later minute bucket, this may be unacceptable for an interactive application.

A relatively small percentage of delayed requests can represent a serious service-quality problem at scale.

Finding 3: Prewarming protects the burst at low additional cost

The third policy reserves extra capacity shortly before the predictable promotion.

In this experiment, prewarming costs only $0.80 more per day than reactive scaling while almost entirely eliminating the modeled burst backlog.

The added expense is small because the extra capacity runs for only a limited interval.

This illustrates a useful principle:

Spend a little more where capacity prevents expensive latency problems, and spend much less where demand does not justify it.

Predictive scaling is particularly valuable for scheduled campaigns, batch submission windows, recurring business peaks, and known model deployment events.

Important interpretation of the latency numbers

The simulator measures queue waiting in one-minute buckets.

A reported wait of zero means that a request was completed within its arrival-minute bucket, not that it completed in zero milliseconds.

Similarly, its p95 and p99 values are not production HTTP latency percentiles.

A production-grade simulation would model service-time distributions, subsecond scheduling, token processing, batching, and end-to-end timing. It would also calculate both queue waiting and execution latency.

The current experiment is suitable for understanding capacity economics and startup delays. It must not be used to certify an interactive latency SLO.

Step 5: Convert the experiment into monthly cost projections

If the same 24-hour demand profile repeats for a 730-hour billing month, the modeled costs become:

The prewarmed policy saves approximately $4,119.87 per modeled month relative to fixed provisioning.

That reduction assumes instance costs scale proportionally with the modeled GPU-hours and that the underlying instances are actually deprovisioned.

It excludes volume storage, networking, other cluster costs, discounts, committed spending, and any unused-capacity penalties imposed by commercial agreements.

For a real system, derive the monthly financial projection from cloud billing records rather than from hardware utilization alone.

Step 6: Experiment with the controller

The most valuable part of the simulation is changing its assumptions.

Try modifying:

• STARTUP_MINUTES from 4 to 8 to represent slow model loading.

• TARGET_UTILIZATION from 0.70 to 0.85 to explore more aggressive consolidation.

• MIN_GPUS from 2 to 1 to reduce quiet-hour spending.

• SCALE_DOWN_COOLDOWN from 15 to 30 to reduce scaling churn.

• The promotional traffic spike to simulate larger or longer bursts.

Watch how the cost savings change alongside queue accumulation.

The experiment should make one important result clear: the lowest compute bill is not necessarily the best deployment policy.

A policy is successful only when it satisfies its service objectives and operating constraints.

5. Translate the scaling strategy into Kubernetes

The simulation demonstrates the economic model. The next challenge is implementing equivalent behavior in a production serving platform.

A common architecture combines application-level metrics, Kubernetes workload scaling, and GPU-node provisioning.

Reference serving architecture:

Clients / API gateway        incoming requests, queue depth, traffic metrics
        |
        v
Prometheus                   stores application and infrastructure metrics
        |
        v
KEDA + HPA                   calculates replica requirements
        |
        v
GPU inference deployment     scales serving replicas from 2 to 10
        |  (unschedulable pods)
        v
GPU node autoscaler          provisions and consolidates cloud GPU nodes

Conceptual control flow. Application metrics feed scaling decisions; unschedulable GPU pods trigger node provisioning.

The critical distinction is that scaling Pods and scaling GPU nodes are separate operations.

A Horizontal Pod Autoscaler can request more model-serving replicas, but those replicas need actual GPU devices on which to run.

Likewise, removing unnecessary replicas does not automatically guarantee that their underlying cloud instances disappear.

Step 1: Install the GPU scheduling components

A Kubernetes GPU deployment generally requires:

• GPU drivers compatible with the nodes and container runtime.

• A vendor device plugin or suitable GPU management operator.

• Nodes advertising GPU resources to the Kubernetes scheduler.

• A Deployment with GPU resource limits.

• A functioning model-serving container with health and readiness probes.

For NVIDIA GPUs, the device resource commonly appears as nvidia.com/gpu.

The current Kubernetes GPU scheduling documentation specifies that GPU limits can be declared without explicit GPU requests because the limit is used as the request. If both are declared, their values must match.

For example, this resource specification belongs inside a GPU-serving container definition:

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    cpu: "4"
    memory: "12Gi"
    nvidia.com/gpu: 1

This is a container resource fragment, not a complete Deployment manifest. Insert it into an existing model-serving Deployment that has the required image, probes, and storage configuration.

The actual CPU and RAM values should come from profiling rather than being treated as defaults.

Step 2: Use KEDA with Prometheus metrics

KEDA provides event-driven workload scaling and integrates with the Kubernetes Horizontal Pod Autoscaler.

Its Prometheus scaler can calculate replica requirements using application metrics.

For our example, assume:

• The namespace is named inference.

• The existing serving Deployment is named ranking-gpu.

• Prometheus is reachable at http://prometheus.monitoring.svc.cluster.local:9090.

• The API gateway exports inference_requests_total, a counter of accepted requests.

• The gateway also exports inference_pending_requests, a gauge representing waiting requests.

• Both metric series have a service="ranking-gpu" label.

• Each GPU replica can safely process 0.7 requests per second under the tested SLO.

These metric names are explicitly defined application instrumentation, not Kubernetes or NVIDIA built-in metrics. They must exist in your monitoring setup.

Create gpu-scaledobject.yaml:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ranking-gpu-autoscaler
  namespace: inference
spec:
  scaleTargetRef:
    name: ranking-gpu

  minReplicaCount: 2
  maxReplicaCount: 10

  pollingInterval: 15

  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300

  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        query: 'sum(rate(inference_requests_total{service="ranking-gpu"}[2m])) or vector(0)'
        threshold: "0.7"
        ignoreNullValues: "false"

    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        query: 'sum(inference_pending_requests{service="ranking-gpu"}) or vector(0)'
        threshold: "120"
        ignoreNullValues: "false"

With KEDA installed, and after confirming that the referenced namespace, Deployment, Prometheus service, and metrics exist, apply the configuration:

kubectl apply -f gpu-scaledobject.yaml

kubectl get scaledobjects -n inference
kubectl get hpa -n inference
kubectl get pods -n inference

Understand what the configuration does

The first Prometheus trigger measures incoming requests per second over a two-minute window.

Its threshold of 0.7 requests per second represents the assumed safe throughput of one GPU replica.

The second trigger reacts to queue depth. A value of 120 means the scaling controller targets approximately 120 queued requests per replica for that metric.

Kubernetes HPA combines the recommendations from multiple metrics by using the largest recommended replica count, subject to configured limits.

The queue threshold is an illustrative starting point, not a universal queueing rule. In a latency-sensitive service, an age-based or token-workload metric may be more informative than raw queue count.

Other important details:

• minReplicaCount: 2 maintains two warm replicas.

• maxReplicaCount: 10 prevents unbounded replica growth.

• The five-minute scale-down stabilization window discourages premature reductions.

• The Prometheus queries return a zero-valued vector when matching metrics are absent.

• The ignoreNullValues: "false" setting asks KEDA to treat missing Prometheus results as errors rather than silently accepting an empty response.

A zero-valued metric is not the same as monitoring health. Production monitoring should separately alert on stale or missing telemetry.

Because the minimum replica count is greater than zero, this example does not use scale-to-zero. Also, KEDA’s polling interval does not directly define the entire HPA reconciliation cadence for scaling between positive replica counts.

This manifest represents a practical starting point, not an exact implementation of the simulation’s custom controller or predictive prewarming behavior.

Step 3: Scale the underlying GPU nodes

Once an HPA increases replicas, Kubernetes needs somewhere to schedule them.

Use a compatible node autoscaler such as Cluster Autoscaler or Karpenter to provision GPU-backed nodes when required and consolidate unneeded nodes as replicas disappear.

The current Kubernetes node autoscaling reference describes this separation between workload scaling and node provisioning.

Several configuration details matter:

• Ensure GPU instance types are allowed by the node provisioning configuration.

• Use GPU-specific node labels and placement rules where appropriate.

• Avoid packing incompatible workloads onto the same GPU nodes.

• Check cloud quotas and accelerator availability before relying on burst scaling.

• Confirm that empty GPU nodes are eventually terminated.

• Account for node termination delays and disruption policies.

A common failure mode is successfully scaling the Deployment from ten replicas to two while leaving the same number of GPU VMs online.

In that situation, Pod utilization appears improved, but the infrastructure bill changes little or not at all.

Step 4: Treat readiness as a real capacity boundary

A newly scheduled model-serving Pod should not be treated as ready merely because its container process started.

A proper readiness signal should indicate that the server can actually accept the required inference traffic.

Depending on the runtime, this may require:

• Model weights fully loaded.

• GPU memory allocations completed.

• The inference engine initialized.

• Required model workers and routing registrations established.

• Internal queues and dependencies functioning.

A readiness probe that reports success prematurely can route requests into a server that still needs significant initialization time.

During scale-down, use appropriate termination grace periods and request draining to avoid abandoning in-flight inference work. Stateful and streaming generation workloads may require careful routing and cancellation handling.

6. Advanced strategies for lowering GPU-serving costs

Autoscaling controls the quantity of resources. Other optimizations improve how efficiently each resource serves useful work.

Dynamic batching

Dynamic batching combines multiple compatible inference requests into GPU execution batches.

This can increase throughput by improving accelerator occupancy and amortizing per-request execution overhead.

However, batching is not free.

Waiting for a larger batch adds scheduling latency, and oversized batches can increase memory pressure or reduce responsiveness.

The appropriate configuration depends on the workload:

• Offline inference can often tolerate larger batches.

• Interactive classification requires tighter batching delays.

• Text generation benefits from iteration-level scheduling and continuous batching.

• Long-context requests need careful KV-cache management.

Benchmark both throughput and latency across batch sizes. Choose the configuration that provides the best SLO-compliant capacity, not merely the highest benchmark requests per second.

Model quantization

Quantization can reduce memory consumption and, on suitable hardware and kernels, improve throughput.

Potential benefits include:

• Hosting a model on a smaller GPU.

• Increasing concurrent request capacity.

• Reducing cross-device communication.

• Supporting additional KV-cache capacity.

However, lower numerical precision does not guarantee faster inference on every architecture.

Compare quality, TTFT, generation speed, memory use, and complete serving cost before migrating a production workload.

Share GPUs where isolation requirements permit

For small models with low traffic, dedicating an entire GPU to each serving replica may be wasteful.

Possible alternatives include:

• Hosting multiple models in a single inference process.

• GPU time-sharing supported by the platform.

• Fractional GPU offerings from the infrastructure provider.

• Hardware partitioning on compatible accelerators.

These approaches differ substantially in memory isolation, performance predictability, supported hardware, and operational complexity.

NVIDIA Multi-Instance GPU, for example, should only be considered on supported GPU architectures. Do not assume all NVIDIA accelerators support the same partitioning capabilities.

GPU sharing also changes the economic unit: multiple Pods may consume one billable GPU, so cost attribution should follow the actual physical or cloud allocation.

Separate interactive and asynchronous inference

Not every request needs the same latency guarantee.

Consider routing traffic into two capacity pools.

Interactive pool

• Maintains a warm minimum.

• Uses conservative utilization targets.

• Prioritizes predictable latency.

• Avoids routine exposure to interruptible capacity when retries are costly.

Asynchronous pool

• Processes durable queued work.

• Can tolerate longer startup and queueing delays.

• Can use more aggressive scaling.

• May benefit from Spot or other interruptible capacity when jobs support retries and checkpointing.

This separation prevents inexpensive batch work from competing with user-facing requests for scarce GPU capacity.

It also allows the system to pursue different optimization goals without overcomplicating a single global autoscaler.

Scale to zero selectively

Scale-to-zero can eliminate compute charges during sufficiently long idle intervals if the infrastructure is actually deprovisioned.

But the benefits depend on the economics of resuming service.

Total return-to-service time can include node provisioning, image download, model weight loading, runtime initialization, and readiness.

If that takes several minutes, sending interactive users directly to a cold endpoint may be unacceptable.

Scale-to-zero is often a better fit for development environments, scheduled batch inference, infrequently used internal endpoints, and asynchronous jobs.

It also requires a traffic or queue signal that remains observable while no inference replicas are running.

7. Build a complete cost model rather than comparing hourly GPU prices

The simulator modeled GPU instance charges. Real cloud bills contain more components.

A practical serving cost model is:

Total monthly cost = compute + storage + network + platform + observability + operational overhead

The associated unit economics are:

Cost per completed request = total attributable cost / successfully completed requests

For a multi-tenant model-serving platform, costs should be allocated according to defensible usage measures, such as model-specific worker time, GPU resource occupancy, or token workload.

Distinguish allocation cost from avoidable cost

This distinction is essential when applying autoscaling to reserved or committed infrastructure.

Suppose an organization is already obligated to pay for eight GPUs under a contract.

Reducing the active deployment to four GPUs may free capacity, but it might not reduce the immediate cash bill.

The savings appear only if the organization can:

• Release uncommitted capacity.

• Reallocate committed capacity to other useful work.

• Reduce future renewals.

• Avoid additional purchases.

• Reduce eligible supporting infrastructure expenses.

Therefore, report two figures whenever possible:

Allocated cost reduction: The accounting value of infrastructure resources no longer assigned to the workload.

Avoidable cash savings: The actual reduction in current or future spending under the organization’s contracts.

Conflating the two can make an optimization look more successful financially than it really is.

Include the cost of failed requests

Lowering replica counts might reduce compute spending while increasing service failures.

Consider a simplified example:

The aggressive policy looks cheapest per successful request, but it loses one million successful requests relative to the fixed-capacity baseline.

Whether that is acceptable depends on user impact, revenue, contractual penalties, and the application’s SLO.

A model-serving cost dashboard should therefore never report cost per request without also reporting success rate and latency.

8. Production failure modes and safeguards

A good autoscaler needs to handle more than ordinary traffic changes.

Sudden bursts

Failure: Demand increases faster than GPUs become ready.

Safeguards: Maintain warm capacity, prewarm predictable peaks, route overflow where supported, and apply admission control when overloaded.

GPU capacity shortages

Failure: Additional Pods remain Pending because suitable GPU nodes are unavailable.

Safeguards: Monitor provisioning time, maintain compatible instance alternatives, understand quotas, and avoid assuming that requested capacity is guaranteed.

Scaling oscillation

Failure: The controller frequently adds and removes GPU replicas.

Safeguards: Apply scale-down stabilization, demand smoothing, minimum replica lifetimes, and realistic controller evaluation intervals.

Monitoring failures

Failure: Metrics become stale or unavailable, preventing sensible scaling decisions.

Safeguards: Alert on telemetry health, establish a conservative fallback capacity, and test what happens when the metrics service is unreachable.

GPU memory exhaustion

Failure: Higher concurrency increases KV-cache or workspace usage beyond available memory.

Safeguards: Enforce concurrency limits, monitor memory headroom, use admission control, and benchmark maximum safe context lengths.

Streaming requests interrupted during scale-down

Failure: Terminating a model server disconnects clients or loses partial outputs.

Safeguards: Stop accepting new requests, drain existing work, configure appropriate termination periods, and test long-running requests.

Infrastructure that refuses to disappear

Failure: Pods terminate but instances, persistent requests, or associated resources continue generating costs.

Safeguards: Audit resource ownership, node consolidation, persistent provisioning requests, and actual billing records.

The persistent Spot request trap illustrates why resource-lifecycle monitoring must extend beyond Kubernetes metrics. Billing alerts and anomaly detection are useful controls, but they can have delays and are not substitutes for correctly terminating billable infrastructure.

9. A practical rollout plan

Rather than immediately activating aggressive scaling in production, introduce optimizations in stages.

Phase 1: Establish the baseline

Collect at least one representative business cycle of operational data.

Measure:

• Hourly traffic and burst characteristics.

• Model-level throughput and token distributions.

• GPU instance-hours and total billed compute.

• Queue depth, age, and cancellation behavior.

• TTFT and end-to-end response latency.

• Readiness and model-loading duration.

• Errors, timeouts, and retries.

Identify the current per-request cost and the fraction of spending associated with idle or unnecessary capacity.

Phase 2: Benchmark alternative deployment shapes

Test smaller GPUs, quantized variants, batching configurations, and different concurrency limits.

Construct a performance matrix that includes cost and application latency rather than relying only on hardware specifications.

Reject configurations that cannot satisfy quality, memory, or reliability requirements.

Phase 3: Replay production traffic

Use historical request-rate and token-demand traces as inputs to a scaling simulation.

Inject realistic disruptions:

• Sudden traffic spikes.

• GPU-node provisioning delays.

• Model-loading failures.

• Metrics outages.

• Interruptible node termination.

• Unbalanced request-length distributions.

Calculate projected GPU-hours, queue delays, failure counts, and costs.

Phase 4: Run a limited production experiment

Begin with one model or a small traffic share.

Keep existing capacity available as a rollback option and establish guardrails for minimum replicas, maximum replicas, and scaling rate.

Compare the new and existing deployment using the same performance and cost definitions.

Phase 5: Verify savings against billing

Measure actual instance-hours, charges, commitments, and shared infrastructure expenses.

Confirm that scaling down Pods leads to fewer billable GPU nodes when expected.

Track the difference between projected and realized savings, then tune startup assumptions, scaling thresholds, and minimum capacity accordingly.

Phase 6: Make optimization continuous

Traffic patterns, model sizes, runtime versions, and cloud prices change.

Rebenchmark after significant model upgrades, GPU migrations, batching changes, or context-length increases.

Review unit economics regularly, not just during the first deployment.

10. Final best-practice checklist

Before considering a GPU-serving platform cost-efficient, confirm the following:

• Models are sized using weights, KV cache, runtime overhead, and representative concurrency.
• Throughput benchmarks include application latency requirements.
• Scaling decisions use workload demand and queues, not GPU utilization alone.
• Startup delays and warm capacity are accounted for.
• Scale-down behavior avoids unnecessary churn and drains in-flight requests.
• GPU node provisioning and consolidation work alongside Pod scaling.
• Scale-to-zero is limited to workloads that can tolerate cold starts.
• Billing measurements verify that infrastructure savings are real.
• Persistent cloud resource requests and orphaned compute are monitored.
• Rollback plans, observability, and load tests are in place.

Conclusion: optimize for productive inference, not maximum GPU usage

GPU right-sizing and autoscaling are most effective when treated as a single capacity-engineering problem.

Right-sizing determines how efficiently each accelerator can serve a workload. Autoscaling determines how much of that capacity should be available at a given moment. Node lifecycle management determines whether reduced demand actually translates into reduced bills.

The simulation demonstrated that fixed provisioning could be substantially more expensive than load-aware scaling under variable traffic. It also exposed the latency risk created by GPU startup delays and showed how scheduled prewarming can mitigate predictable bursts.

The exact savings will vary with model architecture, throughput, traffic, cloud pricing, and service requirements. No scaling policy should be accepted solely because its compute cost is lower.

Your next step: Run the simulation, replace its synthetic demand with a representative anonymized traffic trace, and benchmark your real model at several GPU sizes and concurrency levels. Then deploy a guarded scaling experiment, measure SLO compliance, and compare actual cloud bills before expanding it across your inference fleet.

The objective is simple: pay for the GPU capacity your application genuinely needs, when it needs it, without making users pay the price in latency or reliability.