Inference observability: latency percentiles, token metrics, and drift at the endpoint

An inference endpoint can be healthy by every conventional infrastructure metric and still deliver a terrible user experience.

CPU utilization is normal. Memory usage is stable. HTTP error rates are close to zero. Average request latency is comfortably below the service-level objective (SLO).

Yet some users are waiting ten seconds for a response that usually arrives in 200 milliseconds.

This is the central problem of inference observability: operational health does not necessarily imply acceptable inference performance.

Modern inference workloads introduce additional complexity beyond ordinary HTTP services. Request duration depends on input length, output length, batching, queueing, model execution, hardware utilization, cache behavior, and, for generative models, the speed at which output tokens are produced.

A meaningful observability system must answer several questions simultaneously:

• Latency: What are the p50, p95, and p99 response times?

• Throughput: How many requests and tokens are processed per second?

• Token efficiency: How quickly does generation begin, and how quickly are subsequent tokens produced?

• Reliability: How many requests fail, time out, or exceed the latency budget?

• Model behavior: Are input characteristics, output distributions, and performance patterns changing?

• Root cause: Is a regression caused by the model, serving infrastructure, application code, or workload composition?

In this tutorial, we will build a runnable Python inference service, instrument it with Prometheus-compatible metrics, calculate latency percentiles, generate controlled tail-latency regressions, and design an endpoint-level drift and SLO monitoring strategy.

The implementation uses a deterministic simulation rather than a GPU-dependent language model, so the instrumentation can be tested on an ordinary development machine. We will also identify exactly what must change when connecting the instrumentation to a real inference engine.

1. Why average inference latency is not enough

Suppose an inference endpoint handles 10,000 requests.

Of those requests:

• 9,900 complete in 100 ms.

• 100 complete in 5,000 ms.

The average latency is:

Mean latency = (9,900 × 100 + 100 × 5,000) / 10,000 = 149 ms

A dashboard displaying only average latency reports 149 ms. If the operational target is 200 ms, the service appears healthy.

But the actual distribution tells a different story.

This example highlights an important nuance: when precisely 1% of requests are slow, the p99 boundary lies at the transition between the fast and slow populations. Under the nearest-rank definition, p99 remains 100 ms; interpolation and histogram estimation can produce different values.

Therefore, even p99 is not a complete substitute for measuring the fraction of requests exceeding a user-facing latency threshold.

Understanding p50, p95, and p99

Latency percentiles describe the distribution of request completion times.

p50 — median latency

Half of requests complete at or below this value. It represents the typical experience.

p95 — upper-tail latency

Approximately 95% of requests complete at or below this value. It helps identify degradation affecting a meaningful minority.

p99 — deep-tail latency

Approximately 99% of requests complete at or below this value. It helps reveal expensive outliers, queueing, retries, cold starts, and resource contention.

For an endpoint serving 1,000 requests per second, the slowest 1% represents approximately 10 requests per second.

That can be 600 degraded requests per minute even while the median remains stable.

Why percentiles cannot be averaged

Imagine two inference replicas:

• Replica A serves 9,000 requests with p99 of 150 ms.

• Replica B serves 1,000 requests with p99 of 2,000 ms.

Averaging their p99 values produces 1,075 ms.

That number is not the p99 of the combined 10,000 requests.

Percentiles are not additive. You must merge the underlying request distributions and calculate the percentile afterward.

This is one of the main reasons Prometheus histograms are especially valuable. Histograms export bucket counts that can be aggregated across replicas before calculating percentiles. The official Prometheus documentation describes this aggregation model and the interpolation involved in histogram quantiles.

2. Define the inference measurements before writing instrumentation

A production inference system needs more than one latency measurement.

For a conventional REST prediction endpoint, total request duration may be sufficient as an initial service-level indicator.

For a generative model that streams tokens, it is not.

Request duration

Request duration measures elapsed time from a defined request-processing start to completion or failure.

For the application-level measurement in our implementation:

Request duration = request completion timestamp − application entry timestamp

This includes time spent inside the application, but may exclude connection setup, ingress buffering, and time spent waiting before application code is scheduled.

For a user-facing SLO, an additional measurement from the client or gateway is often necessary.

Time to first token (TTFT)

For streaming inference:

TTFT = first output token timestamp − request start timestamp

TTFT determines how quickly a user sees generation begin.

A model can generate tokens rapidly once started while still having poor TTFT because of queueing, prefill computation, or cold initialization.

Inter-token latency and output throughput

After the first output token, measure the time between subsequent tokens.

For a response producing N output tokens:

Mean time per subsequent token = (last token timestamp − first token timestamp) / (N − 1)

This quantity is defined for N greater than one.

Output token throughput can be calculated from the inverse relationship, but it must be interpreted carefully. Per-request decode throughput and server-wide aggregate token throughput answer different questions, particularly with continuous batching.

Input and output token counts

Input and output token counts describe workload size.

A request with a 12,000-token context should not necessarily be compared directly with a request containing 100 input tokens.

Similarly, generating 2,000 output tokens naturally takes longer than generating 20, assuming the same execution conditions.

We therefore want token counts alongside request latency, not merely as billing statistics.

The initial instrumentation contract

The metric names in our example will be application-specific Prometheus names. They are not presented as OpenTelemetry standard names.

OpenTelemetry’s generative AI semantic conventions provide a separate interoperability layer, including standardized model-operation, token-usage, and inference-duration measurements. Those conventions have continued to evolve during 2026, so instrumentation should follow a clearly selected convention version rather than mix names from different revisions.

3. Build the instrumented inference server

Our reference architecture has four components:

1. A FastAPI inference endpoint.

2. The official Prometheus Python client library.

3. A Prometheus server that periodically scrapes metrics.

4. PromQL queries and recording rules that transform histogram buckets into useful operational signals.

We will expose two HTTP endpoints relevant to inference:

• /infer — accepts inference requests and records telemetry.

• /metrics — exposes Prometheus-compatible metrics.

A third endpoint, /healthz, provides a minimal health check.

Project structure

Create a directory with the following files:

inference-observability/
├── app.py
├── load.py
├── requirements.txt
├── Dockerfile
├── compose.yml
├── prometheus.yml
└── rules.yml

Install the dependencies

We will use FastAPI, Uvicorn, and the official Prometheus client.

The following versions are available as of October 2026 and form a reproducible starting point:

fastapi==0.143.2
uvicorn==0.54.0
prometheus-client==0.26.0

Save this as requirements.txt.

The Python and ASGI packages are supported by actively maintained projects. Prometheus 3.15.0, released in September 2026, will provide the metrics backend for the example.

If you want to run the API without Docker, create a virtual environment and install the dependencies:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

On Windows PowerShell, activate the environment using:

.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Implement the inference endpoint

Create app.py:

import asyncio
import math
import time
from collections import deque

from fastapi import FastAPI
from fastapi.responses import Response
from pydantic import BaseModel, Field
from prometheus_client import (
    CONTENT_TYPE_LATEST,
    Counter,
    Gauge,
    Histogram,
    generate_latest,
)

app = FastAPI(title="Instrumented inference demo")

LATENCY_BUCKETS = (
    0.01, 0.025, 0.05, 0.075, 0.1, 0.15,
    0.25, 0.5, 1.0, 2.0, 4.0, 8.0,
)

REQUESTS = Counter(
    "inference_requests",
    "Completed inference HTTP requests",
    ["outcome"],
)

DURATION = Histogram(
    "inference_request_duration_seconds",
    "Application-level duration of inference HTTP requests",
    ["outcome"],
    buckets=LATENCY_BUCKETS,
)

INPUT_TOKENS = Counter(
    "inference_input_tokens",
    "Simulated input tokens processed",
)

OUTPUT_TOKENS = Counter(
    "inference_output_tokens",
    "Simulated output tokens produced",
)

INPUT_SIZE = Histogram(
    "inference_input_tokens_per_request",
    "Distribution of simulated input token counts",
    buckets=(1, 4, 8, 16, 32, 64, 128, 256, 512),
)

OUTPUT_SIZE = Histogram(
    "inference_output_tokens_per_request",
    "Distribution of simulated output token counts",
    buckets=(1, 4, 8, 16, 32, 64, 128, 256),
)

FIRST_TOKEN = Histogram(
    "inference_internal_first_token_seconds",
    "Time until the first simulated token is ready (not client TTFT)",
    buckets=LATENCY_BUCKETS,
)

TIME_PER_TOKEN = Histogram(
    "inference_time_per_subsequent_token_seconds",
    "Mean interval between successive simulated output tokens",
    buckets=(0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.25, 0.5),
)

IN_FLIGHT = Gauge(
    "inference_inflight_requests",
    "Inference requests being processed",
)

RECENT_LATENCY = Gauge(
    "inference_recent_request_latency_seconds",
    "Exact nearest-rank latency quantiles over the last 2000 successful requests",
    ["quantile"],
)

recent_successful_latencies = deque(maxlen=2000)


class InferenceRequest(BaseModel):
    prompt: str = Field(min_length=1, max_length=4000)
    max_output_tokens: int = Field(default=8, ge=1, le=64)
    slow: bool = False  # Fault injection for the local demo only.


@app.middleware("http")
async def observe_inference_requests(request, call_next):
    if request.url.path != "/infer":
        return await call_next(request)

    start = time.perf_counter()
    outcome = "server_error"
    IN_FLIGHT.inc()

    try:
        response = await call_next(request)

        if response.status_code < 400:
            outcome = "success"
        elif response.status_code < 500:
            outcome = "client_error"

        return response

    finally:
        elapsed = time.perf_counter() - start

        REQUESTS.labels(outcome=outcome).inc()

        DURATION.labels(outcome=outcome).observe(elapsed)

        if outcome == "success":
            recent_successful_latencies.append(elapsed)

        IN_FLIGHT.dec()


@app.post("/infer")
async def infer(body: InferenceRequest):
    begin = time.perf_counter()

    # One whitespace-separated word equals one demo token.
    # Real inference must use tokenizer or provider usage data.
    input_count = len(body.prompt.split())

    INPUT_TOKENS.inc(input_count)
    INPUT_SIZE.observe(input_count)

    # Simulated queueing and prefill delay.
    prefill_delay = 0.03 + 0.001 * min(input_count, 100)

    if body.slow:
        prefill_delay += 1.2

    await asyncio.sleep(prefill_delay)

    vocabulary = (
        "order", "accepted", "and",
        "queued", "for", "processing",
    )

    generated = []
    first_time = None

    for i in range(body.max_output_tokens):
        if i > 0:
            await asyncio.sleep(0.01)

        generated.append(vocabulary[i % len(vocabulary)])

        if first_time is None:
            first_time = time.perf_counter()

    final_time = time.perf_counter()

    FIRST_TOKEN.observe(first_time - begin)

    if len(generated) > 1:
        TIME_PER_TOKEN.observe(
            (final_time - first_time) / (len(generated) - 1)
        )

    OUTPUT_TOKENS.inc(len(generated))
    OUTPUT_SIZE.observe(len(generated))

    return {
        "model": "simulated-v1",
        "text": " ".join(generated),
        "usage": {
            "input_tokens": input_count,
            "output_tokens": len(generated),
        },
        "simulated": True,
    }


@app.get("/metrics")
async def metrics():
    # Local teaching aid: last 2000 successful requests.
    # These quantiles must not be averaged across replicas.
    if recent_successful_latencies:
        ordered = sorted(recent_successful_latencies)

        for quantile in (0.50, 0.95, 0.99):
            index = max(
                0,
                math.ceil(quantile * len(ordered)) - 1,
            )

            RECENT_LATENCY.labels(
                quantile=str(quantile)
            ).set(ordered[index])

    return Response(
        generate_latest(),
        media_type=CONTENT_TYPE_LATEST,
    )


@app.get("/healthz")
async def healthz():
    return {"ok": True}

This is a functioning HTTP service, not merely a collection of instrumentation pseudocode.

It creates valid Prometheus metric families and updates them when requests are processed.

Important distinction: simulated tokens versus model tokens

The demo uses whitespace-separated words as input tokens and emits one predefined vocabulary element per output token.

This is an intentionally simple, deterministic tokenization scheme belonging to the simulated model. It is not equivalent to tokenization used by a production LLM.

When integrating a real model, obtain token counts from:

• The serving engine’s tokenizer and generated token IDs.

• The model server’s response usage metadata.

• The provider’s reported input and output token counts, where available.

Do not assume the number of words equals the number of LLM tokens.

The simulation also measures the time until its first internal token is ready. Because /infer returns a complete JSON response rather than streaming, that metric is not a client-observed time to first token.

This distinction matters. A correct streaming TTFT metric must measure the first token or chunk reaching the chosen observation boundary.

Why use a monotonic clock?

The implementation uses Python’s time.perf_counter().

This clock is suitable for measuring elapsed durations without depending on wall-clock changes.

A wall-clock timestamp is useful for correlating logs and traces. It is not the preferred mechanism for subtracting short timing intervals.

Why instrument the middleware?

The middleware records the application-level duration of every request to /infer, including validation errors and failures that propagate through the request handler.

It records:

• A request counter classified by outcome.

• A latency histogram classified by outcome.

• The number of requests currently being processed.

• A bounded sample of successful request durations.

The outcome label has a fixed, small vocabulary: success, client_error, and server_error.

Keeping this label bounded is intentional. Adding user IDs, request IDs, raw URLs, or prompts as metric labels would create dangerous time-series cardinality.

4. Understand what the server exports

Start the service locally:

uvicorn app:app --host 127.0.0.1 --port 8000

From a second terminal, submit a prediction request:

curl -sS -X POST http://127.0.0.1:8000/infer \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "process this retail order",
    "max_output_tokens": 6
  }'

The JSON response has the following structure:

{
  "model": "simulated-v1",
  "text": "order accepted and queued for processing",
  "usage": {
    "input_tokens": 4,
    "output_tokens": 6
  },
  "simulated": true
}

Now inspect the metrics endpoint:

curl -sS http://127.0.0.1:8000/metrics

The output includes process and Python runtime metrics in addition to the inference measurements.

To focus on the important counters and rolling percentiles:

curl -sS http://127.0.0.1:8000/metrics |
  grep -E '^(inference_requests_total|inference_input_tokens_total|inference_output_tokens_total|inference_recent_request_latency_seconds)'

After the example request, the relevant token counters will include:

inference_input_tokens_total 4.0
inference_output_tokens_total 6.0

The endpoint also exposes three rolling latency gauges:

inference_recent_request_latency_seconds{quantile="0.5"} 0.12
inference_recent_request_latency_seconds{quantile="0.95"} 0.12
inference_recent_request_latency_seconds{quantile="0.99"} 0.12

These latency values are illustrative. Actual elapsed times depend on the machine and execution conditions.

With only one successful request, all three quantiles equal that request’s duration.

Why export both histograms and percentile gauges?

The tutorial deliberately implements two complementary mechanisms.

The direct gauges provide an easy way to inspect the exact nearest-rank p50, p95, and p99 of the last 2,000 successful requests handled by this process.

They are convenient for local demonstrations.

However, they have important limitations:

• They describe a request-count window, not a fixed time window.

• The samples disappear when the process restarts.

• They are specific to one process.

• They are unsuitable for aggregating across multiple replicas.

• They require sorting samples whenever /metrics is scraped.

• Under low traffic, p99 can be unstable or uninformative.

The histogram is the production-oriented source of truth.

It exports cumulative latency bucket counts, a sum, and an observation count.

Unlike locally computed quantiles, these bucket counts can be aggregated across instances and transformed into meaningful service-wide percentiles using PromQL.

The official Python client documentation explicitly recommends histograms for quantile queries: its Summary implementation tracks count and sum but does not compute quantiles locally.

For a real deployment, prefer histogram-derived recording rules. Remove the rolling gauges if their additional operational value does not justify the complexity.

5. Generate traffic that exposes the tail

A single request is not enough to understand a latency distribution.

We need a repeatable workload containing mostly fast requests and a smaller percentage of slower requests.

Create load.py:

import argparse
import json
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor


def send_request(number, slow_every, long_prompts):
    prompt = " ".join(
        ["checkout"] * (96 if long_prompts else 4)
    )

    data = json.dumps({
        "prompt": prompt,
        "max_output_tokens": 8,
        "slow": (
            slow_every > 0
            and number % slow_every == 0
        ),
    }).encode("utf-8")

    request = urllib.request.Request(
        "http://127.0.0.1:8000/infer",
        data=data,
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    with urllib.request.urlopen(
        request,
        timeout=20,
    ) as response:
        return response.status


def main():
    parser = argparse.ArgumentParser()

    parser.add_argument(
        "--requests",
        type=int,
        default=300,
    )

    parser.add_argument(
        "--slow-every",
        type=int,
        default=25,
    )

    parser.add_argument(
        "--long-prompts",
        action="store_true",
    )

    args = parser.parse_args()

    if args.requests < 1 or args.slow_every < 0:
        parser.error(
            "--requests must be positive; "
            "--slow-every cannot be negative"
        )

    begin = time.perf_counter()

    with ThreadPoolExecutor(max_workers=20) as pool:
        statuses = list(
            pool.map(
                lambda i: send_request(
                    i,
                    args.slow_every,
                    args.long_prompts,
                ),
                range(1, args.requests + 1),
            )
        )

    elapsed = time.perf_counter() - begin

    print(
        f"Completed {len(statuses)} requests "
        f"in {elapsed:.2f}s"
    )

    successes = sum(
        200 <= status < 300
        for status in statuses
    )

    print(f"2xx responses: {successes}")


if __name__ == "__main__":
    main()

The script uses Python’s standard library, so no additional HTTP client package is required.

It sends requests concurrently using a thread pool. The workload is deterministic: every 25th request takes the intentionally slow path.

Experiment A: normal traffic

First, generate requests without injected delays:

python load.py --requests 300 --slow-every 0

The server simulates a brief prefill phase followed by token generation.

Record the p50, p95, and p99 values from /metrics.

Experiment B: introduce a 4% slow population

Now run:

python load.py --requests 300 --slow-every 25

This produces 12 deliberately slow requests out of 300.

The slow requests spend an additional 1.2 seconds in the simulated prefill phase.

The expected effect is:

• p50 remains close to the normal baseline.

• p95 remains near the normal population, depending on scheduling.

• p99 rises sharply.

• The mean increases but can remain below a permissive latency target.

For example, in a local execution of a 100-request variant with every 20th request slowed, the observed rolling statistics were approximately:

The exact values are machine-dependent, but the relative behavior demonstrates the issue.

Most requests are healthy. A meaningful subset is more than an order of magnitude slower.

Because the delay occurs before generation begins, the internal first-token metric also rises for affected requests.

That is a useful diagnostic clue: the regression is associated with queueing or prefill-like work, rather than slow subsequent token generation.

Experiment C: change the input distribution

Run another experiment without the slow fault, but with much longer prompts:

python load.py \
  --requests 300 \
  --slow-every 0 \
  --long-prompts

Each request now contains 96 simulated input tokens instead of four.

This changes both the input-token histogram and the prefill duration.

It demonstrates a different phenomenon: latency can increase because the workload changed, even when the serving implementation did not regress.

Separating workload drift from infrastructure regression will be one of our central operational tasks.

6. Connect the service to Prometheus

Our application already exports metrics. We now need Prometheus to collect them continuously.

For reproducibility, we will use Docker Compose with a single API process and a Prometheus server.

The single-process configuration matters: the rolling percentile gauges depend on local memory. Multiple Python worker processes would each maintain separate samples and require different collection semantics.

Create the Dockerfile

Save the following as Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

Create the Prometheus configuration

Save this as prometheus.yml:

global:
  scrape_interval: 5s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules.yml

scrape_configs:
  - job_name: inference_demo
    metrics_path: /metrics
    static_configs:
      - targets:
          - api:8000

Prometheus will scrape the API every five seconds.

It will also load recording rules, which we will define next.

Create the Compose configuration

Save this as compose.yml:

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"

  prometheus:
    image: prom/prometheus:v3.15.0
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules.yml:/etc/prometheus/rules.yml:ro
    depends_on:
      - api

The host ports are bound to localhost. This is appropriate for a development environment and avoids unnecessarily exposing the example’s metrics and fault-injection endpoint on the network.

Add recording rules for p50, p95, and p99

Create rules.yml:

groups:
  - name: inference-observability
    rules:
      - record: inference:latency_p50_seconds:rate5m
        expr: |
          histogram_quantile(
            0.50,
            sum by (le) (
              rate(inference_request_duration_seconds_bucket{outcome="success"}[5m])
            )
          )

      - record: inference:latency_p95_seconds:rate5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (le) (
              rate(inference_request_duration_seconds_bucket{outcome="success"}[5m])
            )
          )

      - record: inference:latency_p99_seconds:rate5m
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(inference_request_duration_seconds_bucket{outcome="success"}[5m])
            )
          )

      - record: inference:latency_good_ratio:rate5m
        expr: |
          sum(rate(inference_request_duration_seconds_bucket{outcome="success",le="0.5"}[5m]))
          /
          clamp_min(
            sum(rate(inference_requests_total{outcome!="client_error"}[5m])),
            1e-9
          )

      - alert: InferenceLatencyBudgetBurn
        expr: |
          (1 - inference:latency_good_ratio:rate5m > 0.05)
          and
          (sum(increase(inference_requests_total{outcome!="client_error"}[5m])) > 100)
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "More than 5% of eligible inference requests are slow or failing"

The first three recording rules continuously derive approximate latency percentiles from histogram buckets.

Notice the sequence of operations:

1. Calculate the rate of change of each histogram bucket.

2. Sum bucket rates across instances while retaining the le bucket-boundary label.

3. Apply histogram_quantile() to the aggregated distribution.

This is the correct aggregation pattern for classic Prometheus histograms. It also remains useful with Prometheus 3.x, which supports both classic and native histograms.

Start the complete stack

If the standalone Uvicorn process is still running, stop it before starting the containers.

Then execute:

docker compose up --build -d

Open the two services:

• API: http://127.0.0.1:8000/docs

• Prometheus: http://127.0.0.1:9090

In Prometheus, open Status → Target health and verify that the inference_demo target is healthy.

Now run:

python load.py --requests 300 --slow-every 25

The request traffic should appear in Prometheus after the next scrape. Recording rules that use a five-minute rate window need multiple samples before producing meaningful values.

You can query the recorded p99 series directly:

inference:latency_p99_seconds:rate5m

Unlike the process-local rolling gauge, this expression is designed to aggregate histogram data from all replicas scraped under the same metric family.

One additional detail: recording-rule results will normally differ from the direct rolling gauges. They use different time windows and percentile algorithms, and Prometheus classic histograms estimate quantiles from buckets rather than retaining every raw duration.

7. Query inference latency with PromQL

Once the metrics are being collected, we can create practical dashboard queries.

p50, p95, and p99

The recorded series from the previous section can be displayed directly:

inference:latency_p50_seconds:rate5m

inference:latency_p95_seconds:rate5m

inference:latency_p99_seconds:rate5m

For a different observation window, query the original histogram.

For example, p99 across the previous 15 minutes:

histogram_quantile(
  0.99,
  sum by (le) (
    rate(inference_request_duration_seconds_bucket{outcome="success"}[15m])
  )
)

Changing the window changes the statistical population.

A five-minute p99 is useful for detecting short-lived regressions. A 30-minute or one-hour p99 is often more stable but can smooth over brief incidents.

Average latency

Because a histogram also exports sum and count, we can calculate mean latency:

sum(
  rate(inference_request_duration_seconds_sum{outcome="success"}[5m])
)
/
clamp_min(
  sum(
    rate(inference_request_duration_seconds_count{outcome="success"}[5m])
  ),
  1e-9
)

Keep this measurement, but do not make it the sole indicator of health.

It is especially useful when interpreted alongside percentiles. A stable mean with a rising p99 suggests a relatively small slow population. A rising mean and median suggest broader degradation.

Requests per second

sum(
  rate(inference_requests_total[5m])
)

Traffic volume is an essential companion metric to latency.

A p99 increase at 10 requests per minute requires a different investigation from an increase at 10,000 requests per second.

Output tokens per second

sum(
  rate(inference_output_tokens_total[5m])
)

This is the application-wide output token rate.

With multiple serving replicas, adding their output token rates gives total throughput.

That aggregation is mathematically valid because token counts are additive.

Input tokens per second

sum(
  rate(inference_input_tokens_total[5m])
)

The relationship between input throughput, output throughput, and request throughput helps explain performance changes.

For example:

• Stable requests/second with rising input tokens/second indicates larger input workloads.

• Stable requests/second with rising output tokens/second indicates longer generated responses.

• Falling output tokens/second with rising in-flight requests may indicate decode slowdown, queueing, or resource saturation.

These patterns suggest where to investigate; they do not independently establish root cause.

p95 first-token readiness

Our simulator exposes internal first-token readiness:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(inference_internal_first_token_seconds_bucket[5m])
  )
)

In a streaming production server, the equivalent measurement should be connected to a precisely defined streaming boundary.

Time to first token often deserves its own SLO because users perceive the start of generation differently from the time needed to complete a long response.

Output tokens per completed request

sum(
  rate(inference_output_tokens_total[5m])
)
/
clamp_min(
  sum(
    rate(inference_requests_total{outcome="success"}[5m])
  ),
  1e-9
)

In this simulator, token usage is recorded for requests that complete generation successfully.

A real server should carefully distinguish tokens generated, tokens delivered, and billable tokens. Failed or canceled generations may still consume model compute or incur charges, so a production accounting policy must handle partial output and failed operations explicitly.

8. Define an inference latency SLO

Percentiles help describe performance, but service-level objectives provide an actionable reliability contract.

Consider the following objective:

At least 99% of eligible inference requests must complete successfully within 500 milliseconds over a rolling 30-day period.

This means:

• The latency threshold is 500 ms.

• The target success proportion is 99%.

• The measurement population excludes invalid client requests.

• Server failures count as bad requests.

• Slow successful responses also count as bad requests.

The error budget is 1% of eligible requests.

Calculate the good-request ratio

We already defined a five-minute recording rule using this expression:

sum(
  rate(
    inference_request_duration_seconds_bucket{
      outcome="success",
      le="0.5"
    }[5m]
  )
)
/
clamp_min(
  sum(
    rate(
      inference_requests_total{
        outcome!="client_error"
      }[5m]
    )
  ),
  1e-9
)

The numerator counts successful requests that complete within 500 ms.

The denominator counts eligible requests, including server failures.

Our example excludes invalid client requests, but that policy should be explicitly documented. Whether authentication failures, client disconnects, and canceled generations are eligible depends on the service contract.

Understand error-budget burn rate

A 99% SLO allows 1% bad requests.

If 5% of requests are bad, the service is consuming its error budget at five times the sustainable rate.

Burn rate = observed bad-request fraction / allowed bad-request fraction

For example:

Burn rate = 0.05 / 0.01 = 5

The recording rules include a teaching-oriented alert that fires when the five-minute bad-request ratio exceeds 5% for ten minutes, provided the request volume is sufficient.

That is useful for the lab. A production deployment should generally use multiwindow, multi-burn-rate alerting to distinguish brief severe incidents from sustained moderate degradation.

Why percentile alerts alone are insufficient

A p99 alert might fail to detect exactly 1% of requests that are catastrophically slow, depending on the distribution and percentile estimation.

A threshold-based SLO directly counts requests that exceed the acceptable duration.

The two approaches complement each other:

• Percentiles explain the shape of latency.

• Threshold compliance quantifies customer-facing reliability.

• Burn-rate alerts tell operators how quickly the reliability budget is being consumed.

For low-volume endpoints, SLO compliance counts can be more interpretable than a noisy p99.

9. Detect drift at the inference endpoint

Model drift is often discussed as a machine-learning evaluation problem. In production serving, drift monitoring is also an operational necessity.

A model can continue returning predictions successfully while the population of requests changes significantly.

Consider a retail-order summarization service.

At launch, typical requests may contain 100 input tokens and request 50 output tokens.

Several months later, clients begin submitting entire transaction histories. The typical input length rises to 4,000 tokens while output length remains approximately constant.

The service may experience:

• Higher input-processing latency.

• Greater memory consumption.

• More queueing under concurrent load.

• Reduced maximum throughput.

• Higher inference cost.

None of these changes necessarily imply a software or model regression.

Distinguish three categories of drift

1. Workload drift

The distribution of input sizes, request parameters, or application use cases changes.

Examples include longer prompts, different batch sizes, different document types, and increasing context lengths.

1. Output or prediction drift

The distribution of generated outputs, scores, classes, or other model outputs changes.

This may result from input drift, model changes, parameter changes, or changes in external application behavior.

1. Performance drift

Latency, memory demand, throughput, error rates, or token-generation performance changes.

Performance drift can be caused by model behavior, infrastructure, scheduling, or the workload itself.

An observability system should investigate these categories together rather than treating every latency regression as a serving bug.

Use input-token histograms to monitor workload composition

The application already exports:

inference_input_tokens_per_request_bucket

We can calculate p95 input size:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(inference_input_tokens_per_request_bucket[15m])
  )
)

This provides a useful signal for detecting larger input workloads.

For longer-term comparison, calculate a ratio between the current p95 and the equivalent window one day earlier:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(inference_input_tokens_per_request_bucket[15m])
  )
)
/
clamp_min(
  histogram_quantile(
    0.95,
    sum by (le) (
      rate(inference_input_tokens_per_request_bucket[15m] offset 1d)
    )
  ),
  1
)

A ratio of two means the current p95 is approximately twice the value from the comparison window.

This query requires at least one day of historical data and is not immediately useful on a freshly started server.

The comparison window should also account for normal daily and weekly variation. An equivalent period from the previous day is only one possible baseline.

Detect distribution shift with Population Stability Index

Comparing a single quantile can miss changes in the shape of the distribution.

For a richer comparison, we can use Population Stability Index (PSI).

PSI compares the proportions of observations in predefined bins between a reference distribution and a current distribution.

For each bin:

Contribution = (Current proportion − Reference proportion) × ln(Current proportion / Reference proportion)

PSI is the sum of these contributions.

The following example uses synthetic retail request-length distributions.

import math


def population_stability_index(
    reference_counts,
    current_counts,
    epsilon=1e-6,
):
    if len(reference_counts) != len(current_counts):
        raise ValueError("Bin counts must have equal length")

    if not reference_counts:
        raise ValueError("At least one bin is required")

    reference_total = sum(reference_counts)
    current_total = sum(current_counts)

    if reference_total <= 0 or current_total <= 0:
        raise ValueError("Both distributions must be nonempty")

    result = 0.0

    for reference_count, current_count in zip(
        reference_counts,
        current_counts,
    ):
        reference_p = max(
            reference_count / reference_total,
            epsilon,
        )

        current_p = max(
            current_count / current_total,
            epsilon,
        )

        result += (
            (current_p - reference_p)
            * math.log(current_p / reference_p)
        )

    return result


# Fixed bins: 0-32, 33-64, 65-128, and 129+ tokens.
reference = [500, 250, 200, 50]
current = [150, 200, 250, 400]

score = population_stability_index(
    reference,
    current,
)

print(f"Population Stability Index: {score:.4f}")

The reference distribution contains mostly short requests.

The current distribution contains a substantially larger proportion of long requests.

The calculated score reflects that distributional change.

For production use, establish the reference bins using a representative dataset and keep those bins fixed when comparing new observations.

Prometheus histogram buckets can provide the required raw information, but remember that their counts are cumulative. Convert them into nonoverlapping bin counts before calculating PSI.

PSI values should not be interpreted as universal statistical significance tests. Thresholds depend on the reference population, binning strategy, volume, and sensitivity of the application.

What about actual model-quality drift?

Endpoint telemetry can expose changes in input and output distributions, but it cannot by itself establish that model accuracy or response quality has degraded.

For that, collect evaluation signals such as:

• Prediction error when ground truth eventually becomes available.

• Task-completion rates.

• Schema-validation or structured-output failure rates.

• Human-reviewed quality scores.

• Results from a stable evaluation dataset.

• Policy compliance or application-specific correctness tests.

For generative models, quality evaluation frequently requires sampled offline evaluation, delayed feedback, or controlled test prompts.

Keep those measurements separate from request latency. A model that becomes faster but less accurate is not necessarily an improvement.

10. The cherry on the cake: a real production incident where p99 exposed the problem

In August 2026, OpenResty’s Yichun Zhang published an engineering analysis describing several real production tail-latency investigations (the D-language case is covered in detail in an earlier OpenResty post).

One particularly instructive case involved an order-processing service written in the D programming language.

The service’s routine monitoring looked healthy: throughput, error rates, and average latency did not reveal the severity of the problem.

Yet p99 latency increased from approximately 120 ms to 350 ms.

The engineers initially investigated several conventional suspects, including database behavior, external dependencies, and concurrency settings.

Those investigations did not explain the tail.

Profiling eventually identified two important causes inside the service:

• Garbage-collection activity that consumed substantial CPU time.

• An inefficient linear-scan operation whose cost grew under real production workloads.

According to the published analysis, garbage collection accounted for 26.4% of CPU time in the profiling results, while the problematic array-filtering path accounted for 59.4% of CPU.

The exact CPU measurements describe separate identified hot spots; they should not be interpreted as a universal decomposition of all request latency.

The important lesson is that normal-looking aggregate indicators did not establish that the application was performing well. Tail-latency measurement identified a problem that conventional monitoring had not made obvious.

Why this incident matters for inference serving

The same failure pattern can appear in machine-learning systems.

Suppose a model-serving cluster experiences occasional CPU-intensive request preprocessing.

Most requests execute normally, but certain long inputs trigger expensive transformations.

The resulting behavior might be:

• Unchanged model weights.

• Unchanged prediction quality.

• Stable GPU utilization.

• Mostly healthy response times.

• Significantly degraded p99 latency.

An engineer watching only GPU utilization and average response time might incorrectly conclude that the serving layer is healthy.

An engineer watching p99 together with input-size histograms, preprocessing spans, and runtime profiles has a much better chance of discovering the actual bottleneck.

The incident also demonstrates why performance investigation should not begin by assuming that the model computation is responsible.

11. Design production dashboards around diagnosis, not decoration

A good dashboard should help operators move quickly from symptoms to hypotheses.

For inference services, I recommend organizing panels into four groups.

Dashboard A: endpoint reliability

Display:

• Request rate.

• Client and server error rates.

• p50, p95, and p99 request duration.

• Fraction of requests within the latency SLO.

• Error-budget burn rate.

This dashboard answers whether the endpoint is currently meeting its service commitments.

Dashboard B: token-level performance

Display:

• Input tokens per second.

• Output tokens per second.

• Input-token count distribution.

• Output-token count distribution.

• Time to first token.

• Time per subsequent output token.

This dashboard helps distinguish longer workloads from slower processing.

With a genuine streaming server, include separate measurements for request queueing, prefill, decoding, and response delivery when those stages can be observed reliably.

Dashboard C: serving infrastructure

Display:

• In-flight requests.

• Queue depth and queue wait time.

• CPU and accelerator utilization.

• Device memory utilization.

• Model cache usage and cache-hit ratios.

• Batch sizes.

• Request rejection and timeout rates.

The exact infrastructure metrics depend on the serving engine and deployment environment.

CPU usage alone is insufficient, and GPU utilization alone can be equally misleading. A highly utilized GPU may be doing useful work, while a seemingly idle GPU may be waiting for input preparation, scheduling, or data movement.

Dashboard D: workload and model behavior

Display:

• Input-size percentiles.

• Output-size percentiles.

• Request rates by model version.

• Changes in request configuration.

• Distribution-drift indicators.

• Quality evaluation results.

This dashboard helps determine whether latency or error changes coincide with changes in input composition or deployed model behavior.

Use model-version labels carefully

Model version is often a useful metric dimension.

For example, during a canary deployment, separate latency distributions for the existing model and candidate model make regressions easier to spot.

But use stable, bounded identifiers.

Do not turn arbitrary model names, tenant-supplied strings, prompt content, or request-specific identifiers into labels.

A metric family multiplied across hundreds of thousands of unique labels becomes expensive to store, expensive to query, and potentially unreliable.

12. Connect metrics to traces and logs

Metrics identify patterns. Traces help explain where the time went. Logs preserve discrete operational events.

For complex inference systems, these signals work best together.

Imagine p99 suddenly increasing from 800 ms to 3 seconds.

The metrics reveal that the regression affects only certain requests.

The input-token histogram shows that the affected workload contains longer prompts.

A distributed trace can then break down a slow request into stages:

• Gateway and authentication.

• Request validation.

• Retrieval or feature lookup.

• Queue waiting.

• Input processing or prefill.

• Generation.

• Output serialization.

This helps distinguish a serving-engine problem from an application-level dependency.

Current OpenTelemetry conventions

OpenTelemetry’s generative AI semantic conventions provide a structured approach to representing model interactions and usage information.

The 2026 inference conventions include token-usage counters such as gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokens, as well as inference-operation duration measurements.

These conventions are developed separately from the application-specific Prometheus metrics used in our demo. Their definitions and stability status should be checked against the selected specification version before deployment.

A practical integration strategy is:

1. Keep stable Prometheus histograms and counters for operational monitoring.

2. Add OpenTelemetry spans around model inference and critical dependencies.

3. Correlate latency regressions with traced request stages.

4. Export only the attributes needed for diagnosis.

5. Use sampling and access controls for potentially sensitive trace content.

For privacy and performance reasons, avoid recording full prompts and generated responses by default.

Even when such content would help debugging, it can expose customer data, increase storage costs, and create unnecessary security risk.

13. Common instrumentation mistakes

Several problems repeatedly undermine inference observability.

Mistake 1: averaging percentiles

Averaging p99 values across replicas does not produce a fleet-wide p99.

Aggregate histogram buckets first, then calculate the percentile.

Mistake 2: using the wrong latency boundary

Application middleware measures a different interval from an external client.

For a streaming response, the time until the handler returns a response object is not the same as the time until the final byte reaches the client.

If converting this demo to streaming, instrument the actual response-generation lifecycle and handle cancellations and disconnects.

Mistake 3: counting words instead of model tokens

Our simulator deliberately defines its own simple tokenization.

A production model’s tokenizer may split a single word into several tokens or handle whitespace, punctuation, and multilingual text differently.

Use actual token IDs or authoritative usage metadata.

Mistake 4: treating timeout failures as missing data

If timeout and error requests are omitted from the duration distribution, a latency dashboard may improve during an outage because the worst requests no longer appear as successful completions.

Record request outcomes separately, retain appropriate duration measurements for failures, and include failures in the reliability objective.

Mistake 5: using buckets that are too coarse

If most requests finish between 100 and 150 ms but histogram boundaries jump from 100 ms to 500 ms, percentile estimates will lack useful precision.

Choose bucket boundaries around relevant SLO thresholds and the latency distribution you actually observe.

Classic histogram quantiles are estimates, and their accuracy depends on bucket placement.

Mistake 6: ignoring the number of observations

A p99 calculated from a small number of requests can be dominated by one observation.

Always inspect volume alongside percentile changes.

Mistake 7: treating request drift as model degradation

A shift toward longer prompts can increase latency without any change to the model or serving implementation.

Segment by workload properties before concluding that serving performance has regressed.

Mistake 8: mishandling multiple Python workers

The example deliberately uses one Uvicorn worker.

The Prometheus Python client has a dedicated multiprocess mode with special collection requirements and limitations, including restrictions on custom collectors and some gauge behaviors. Simply increasing the number of workers in this demo would not produce correct shared rolling-percentile gauges.

For production deployments, either use correctly configured multiprocess collection or make each worker independently scrapeable. Prefer aggregatable histogram metrics over process-local quantile gauges.

14. Production hardening checklist

Before deploying this observability pattern with a real model, review the following areas.

Measurement correctness

• Define exactly where request timing starts and ends.

• Measure streaming response completion and cancellation correctly.

• Use real model token counts.

• Separate inference queueing, prefill, and decoding when available.

• Include failures in reliability accounting.

Metric design

• Use seconds for duration metrics.

• Choose buckets around meaningful thresholds.

• Keep metric labels bounded.

• Validate histogram aggregation across replicas.

• Avoid treating process-local quantiles as fleet-wide measurements.

Reliability

• Define an explicit eligible-request population.

• Establish latency and error-rate objectives.

• Configure sensible alert windows and minimum traffic volume.

• Verify alerts with fault injection.

• Track request rates alongside percentile changes.

Model monitoring

• Track the deployed model version.

• Track workload-size distributions.

• Compare current metrics with representative historical baselines.

• Monitor prediction or generation quality separately.

• Investigate input and output drift together.

Operational security

• Restrict access to /metrics.

• Remove the demo’s client-controlled slow fault-injection parameter.

• Apply request validation, authentication, and rate limits appropriate to the service.

• Avoid exporting prompts, customer identifiers, or generated content as metric labels.

• Establish retention and access policies for telemetry.

Capacity and scalability

• Measure saturation under realistic concurrency.

• Verify serving behavior during autoscaling.

• Test cold starts and model reloads.

• Monitor queue depth and request rejection.

• Load-test with representative input and output lengths, not only short synthetic requests.

These safeguards turn a useful demonstration into a more reliable production monitoring design.

15. Bringing it all together

Inference observability is not simply about putting a latency graph next to a GPU utilization graph.

It requires connecting user-facing performance to the behavior of the underlying inference workload.

The most useful design principles are:

• Use histograms for aggregatable latency percentiles. Direct process-local quantiles can help during development, but they are not substitutes for fleet-wide distributions.

• Monitor token usage alongside latency. Request count alone is a weak measure of generative inference workload.

• Separate first-token responsiveness from complete-response latency. They reflect different aspects of the serving path.

• Use threshold-based SLOs to measure reliability. Percentiles describe latency distributions; good-request ratios describe compliance with a concrete service objective.

• Monitor workload drift before blaming the model. Longer or more complex requests can produce genuine performance degradation without changes to model execution.

• Connect metrics to traces and evaluation results. No single measurement explains both performance and model quality.

Our runnable server demonstrates these principles without depending on specialized hardware. It exports request durations, p50/p95/p99 learning gauges, token counters, token-size histograms, internal first-token timing, and in-flight request counts. Prometheus then turns the histogram data into queryable latency percentiles, reliability measurements, and alerts.

Your next step

Run the complete stack, generate both normal and fault-injected traffic, and compare p50, p95, p99, and SLO compliance in Prometheus.

Then replace the deterministic generation function with your actual inference engine, use real token accounting, and add the observability boundaries that match your production serving architecture.

The goal is not merely to know that inference is slow. It is to know which requests are slow, how badly they affect the service objective, what changed, and where to investigate first.