Speculative decoding and other tricks that make inference 2x faster

1. Why LLM inference is slower than it needs to be

A modern large language model can process thousands of input tokens in parallel, yet generating its response still involves a surprisingly inefficient process: producing one token at a time.

Consider a model generating the following sentence:

The customer placed an order and received confirmation.

The model cannot ordinarily generate this entire sentence in one operation. It predicts the next token, adds that token to its context, and repeats.

For a response containing 500 tokens, this may require approximately 500 sequential decoding steps.

Even with a powerful GPU, the cumulative latency becomes significant.

The challenge is especially pronounced for:

• Interactive AI assistants that need low response latency.

• Coding agents generating thousands of tokens.

• Reasoning models producing long outputs.

• Retrieval-augmented generation systems.

• Production APIs serving many simultaneous users.

• Applications running large models on bandwidth-constrained hardware.

The interesting part is that simply adding more GPU compute does not always solve this problem.

During low-batch autoregressive decoding, inference is frequently limited by memory bandwidth rather than raw arithmetic throughput. The GPU repeatedly moves model weights and cached attention data through memory, even though relatively little computation may be performed per generated token.

This creates an opportunity: what if we could produce multiple tokens while paying for only one expensive verification operation?

That is the central idea behind speculative decoding.

As of 2026, speculative decoding is part of mainstream inference frameworks, including vLLM and SGLang. Modern implementations support draft models, multi-token prediction, feature-based drafting, and lightweight methods that require no additional language model.

We will explore how these methods work, implement the core algorithm in Python, deploy a practical configuration, and examine the additional optimizations that can make an inference stack substantially faster.

2. Understand the two phases of transformer inference

Before optimizing inference, we need to distinguish two fundamentally different operations.

Prefill: processing the input prompt

Suppose a user submits a prompt containing 2,000 tokens.

During prefill, the transformer processes those tokens and constructs the key-value cache, commonly called the KV cache.

Because the entire input sequence is available, the model can process many positions in parallel using causal attention.

Prefill often benefits from:

• High-throughput matrix multiplication.

• Efficient attention kernels.

• Tensor Core acceleration.

• Large computational batches.

• Reusing previously computed prompt prefixes.

For sufficiently large prompts, prefill can be compute-intensive.

Decode: generating new tokens

Once prefill finishes, the model enters autoregressive decoding.

For each generated token, the model generally:

1. Reads relevant previous attention keys and values.

2. Executes the transformer layers.

3. Produces logits over the vocabulary.

4. Applies decoding rules such as temperature and top-p filtering.

5. Samples or selects a token.

6. Updates the KV cache.

7. Repeats the process.

The important difference is that each step depends on the previously generated tokens.

This prevents ordinary autoregressive generation from parallelizing future token decisions directly.

Why this creates a memory bottleneck

Imagine a hypothetical model with 14 billion parameters stored using two bytes per parameter.

Its parameter weights alone occupy approximately 28 GB, excluding KV cache, activations, runtime buffers, and other overhead.

At low batch sizes, generating a single token may require reading a substantial fraction of those weights.

If model weights dominate memory traffic, the simplified relationship is:

Decode time ≈ bytes transferred / effective memory bandwidth

This is not a complete performance model. Attention, communication, kernel launches, and computation also contribute.

However, it explains why a large GPU with impressive theoretical FLOPS may still struggle to achieve low per-token latency.

It also suggests an optimization strategy: improve the amount of useful output produced per expensive model invocation.

3. Speculative decoding: predict cheaply, verify expensively

Speculative decoding introduces two roles:

• Draft model: A relatively inexpensive predictor that proposes several future tokens.

• Target model: The original model that evaluates those predictions.

The draft model makes educated guesses about the next tokens. The target model then verifies the proposed sequence using a parallelizable forward pass.

Crucially, correctly implemented speculative sampling preserves the target model’s output probability distribution.

The draft model does not replace the target model’s decisions. It helps the target model evaluate several potential decisions more efficiently.

A simple example

Suppose the current prompt is:

The capital of France is

The draft model proposes:

Paris, which is located

The target model evaluates these candidate tokens together.

If the proposed sequence is accepted, the system commits multiple tokens after a single verification operation. If a proposal is rejected, the system corrects the sequence using the target distribution.

Illustrative speculative decoding cycle:

Current prefix:        The capital of France is

Cheap draft proposes:  [Paris] [,] [which] [is]          (4 cheap draft steps)
                           |    |     |     |
One target pass:       [Paris] [,] [which] [is]  -> all 4 accepted
                                                  + 1 bonus token sampled by the target

Conceptual example only. At nonzero temperature, acceptance is probabilistic, not simply a string-matching operation.

Why parallel verification is possible

There is an important distinction between generating tokens and evaluating an already proposed token sequence.

Autoregressive generation requires each new token to be known before selecting the next one.

Verification already has a complete candidate sequence.

The target model can process that candidate block with causal attention, obtaining the conditional probability for each proposed token.

For example, given a prefix and four proposed tokens, the verifier evaluates:

• Position 1 using the original prefix.

• Position 2 using the prefix plus proposed token 1.

• Position 3 using the prefix plus proposed tokens 1 and 2.

• Position 4 using the prefix plus proposed tokens 1 through 3.

The verifier can also produce the probability distribution needed to sample a bonus token when all proposals are accepted.

This single verification pass performs more computation than an ordinary one-token decode step, but can be much cheaper than four or five separate target-model decoding passes.

That difference creates the speedup.

4. The exact acceptance algorithm

A common misconception is that speculative decoding simply accepts a candidate when the target model assigns it sufficiently high probability.

That is not the complete algorithm for exact speculative sampling.

Let:

• p(x) denote the target model’s probability of token x.

• q(x) denote the draft model’s probability of token x.

• y denote a token sampled from the draft distribution.

The acceptance probability is:

A(y) = min(1, p(y) / q(y))

If the proposal is accepted, it becomes part of the output.

If the proposal is rejected, the system does not simply sample directly from p.

Instead, it samples from a residual distribution:

R(x) ∝ max(0, p(x) − q(x))

The residual distribution is normalized to sum to one.

After a rejection, all subsequent unverified draft tokens are discarded.

If every draft token is accepted, the target model can provide one additional token using its own distribution.

A worked probability example

Suppose the vocabulary contains only two tokens: A and B.

The target probabilities are:

• p(A) = 0.70

• p(B) = 0.30

The draft probabilities are:

• q(A) = 0.90

• q(B) = 0.10

The draft model proposes A 90% of the time.

The acceptance probability for A is:

min(1, 0.70 / 0.90) = 7/9

The acceptance probability for B is:

min(1, 0.30 / 0.10) = 1

When A is rejected, the residual distribution assigns all probability to B.

Therefore:

• Final probability of A = 0.90 × 7/9 = 0.70

• Final probability of B = 0.10 + 0.90 × 2/9 = 0.30

The resulting distribution matches the target exactly.

This is why an imperfect draft model can still be useful.

Its predictions affect execution efficiency, but the acceptance mechanism corrects their statistical bias.

What “lossless” actually means

For exact speculative sampling, lossless means preserving the target model’s sampling distribution, subject to correct implementation and numerical behavior.

It does not necessarily mean:

• Identical token sequences under independent random seeds.

• Bit-for-bit equality between different inference kernels.

• Identical behavior after quantizing or otherwise modifying the target.

• Distribution preservation when custom sampling rules are applied inconsistently.

• Automatic correctness for every experimental approximate verification method.

Greedy speculative decoding has a related, simpler property: accepted draft tokens must agree with the target’s greedy choices.

These distinctions matter when comparing inference optimizations in production.

5. Build speculative decoding from scratch in Python

Now let’s implement the algorithm without PyTorch, transformers, GPUs, or external dependencies.

Instead of loading a real language model, we will use a synthetic shopping-event generator.

Its vocabulary consists of four events:

• view

• click

• cart

• buy

We will define two probability models representing the target and draft predictors.

This keeps the demonstration transparent while preserving the essential mathematics of speculative sampling.

Step 1: Prepare the environment

Python 3.10 or newer is sufficient.

python -m venv .venv
source .venv/bin/activate

On Windows, use your shell’s corresponding virtual-environment activation command.

Save the following code as a file named speculative_demo.py.

Step 2: Implement the target and draft distributions

from collections import Counter
from random import Random

TOKENS = ("view", "click", "cart", "buy")

# Synthetic shopping-event transition probabilities.
TARGET = {
    "START": (0.60, 0.25, 0.10, 0.05),
    "view":  (0.30, 0.55, 0.12, 0.03),
    "click": (0.20, 0.25, 0.45, 0.10),
    "cart":  (0.10, 0.10, 0.25, 0.55),
    "buy":   (0.70, 0.20, 0.08, 0.02),
}

DRAFT = {
    "START": (0.58, 0.27, 0.10, 0.05),
    "view":  (0.33, 0.51, 0.13, 0.03),
    "click": (0.22, 0.23, 0.43, 0.12),
    "cart":  (0.12, 0.10, 0.23, 0.55),
    "buy":   (0.66, 0.23, 0.09, 0.02),
}


def probs(model, history):
    previous = TOKENS[history[-1]] if history else "START"
    return model[previous]


def draw(weights, rng):
    threshold = rng.random() * sum(weights)

    for index, weight in enumerate(weights):
        threshold -= weight
        if threshold < 0:
            return index

    return len(weights) - 1


def baseline(n, rng):
    result = []

    for _ in range(n):
        distribution = probs(TARGET, result)
        result.append(draw(distribution, rng))

    return result

The target model determines the desired output distribution.

The draft model approximates that distribution, with slightly different probabilities.

The models are first-order Markov models: each prediction depends on the immediately preceding event. A real transformer instead conditions on a much richer token history.

The baseline function performs ordinary autoregressive generation, sampling one token at a time.

Step 3: Add speculative sampling

Append this function to the same file.

def speculative(n, k, rng):
    result = []

    verification_passes = 0
    proposed = 0
    accepted = 0

    while len(result) < n:
        candidates = []
        draft_distributions = []

        # Phase 1: Draft up to k future tokens.
        for _ in range(min(k, n - len(result))):
            q = probs(DRAFT, result + candidates)
            token = draw(q, rng)

            candidates.append(token)
            draft_distributions.append(q)

        proposed += len(candidates)

        # Phase 2: One conceptual parallel verification.
        verification_passes += 1

        for token, q in zip(
            candidates,
            draft_distributions,
        ):
            p = probs(TARGET, result)

            acceptance = min(1.0, p[token] / q[token])

            if rng.random() < acceptance:
                result.append(token)
                accepted += 1

                if len(result) == n:
                    break
            else:
                # Correct the draft's probability bias.
                residual = [
                    max(0.0, a - b)
                    for a, b in zip(p, q)
                ]

                result.append(draw(residual, rng))

                # Later draft tokens are no longer valid.
                break
        else:
            # Every candidate was accepted.
            # Sample a bonus token from the target.
            if len(result) < n:
                p = probs(TARGET, result)
                result.append(draw(p, rng))

    return result, verification_passes, proposed, accepted

There are several important details.

First, the draft distributions are saved alongside their proposed tokens. We need the original probabilities to compute exact acceptance ratios.

Second, the target distribution is evaluated along the current accepted prefix.

Third, when a proposal is rejected, the replacement comes from the residual distribution—not the draft distribution and not the unmodified target distribution.

Fourth, the algorithm adds a bonus token only when all candidates are accepted and the desired output length has not already been reached.

The target probability lookups appear sequentially in this small Python illustration. In an actual GPU implementation, the verifier obtains the necessary distributions through a single parallelized target-model pass.

Step 4: Compare the output distributions

We should verify more than whether the code runs.

A correct implementation must preserve the target distribution, even though individual sampled sequences differ.

Append the following experiment.

def main():
    rng_plain = Random(101)
    rng_spec = Random(202)

    trials = 20000

    # Compare joint distributions of three events.
    plain_counts = Counter(
        tuple(baseline(3, rng_plain))
        for _ in range(trials)
    )

    spec_counts = Counter(
        tuple(speculative(3, 4, rng_spec)[0])
        for _ in range(trials)
    )

    outcomes = plain_counts.keys() | spec_counts.keys()

    max_gap = max(
        abs(plain_counts[key] - spec_counts[key]) / trials
        for key in outcomes
    )

    print(
        f"Largest 3-token outcome frequency gap: "
        f"{max_gap:.4f}"
    )

    # Measure the number of conceptual target passes.
    test_rng = Random(42)
    runs = 1000

    statistics = [
        speculative(64, 4, test_rng)
        for _ in range(runs)
    ]

    passes = sum(item[1] for item in statistics)
    proposals = sum(item[2] for item in statistics)
    accepted = sum(item[3] for item in statistics)

    print("Baseline target steps per 64 tokens: 64")

    print(
        "Illustrative verification passes per 64 tokens: "
        f"{passes / runs:.2f}"
    )

    print(
        f"Draft acceptance fraction: "
        f"{accepted / proposals:.1%}"
    )

    print(
        "These are algorithmic counts, "
        "not wall-clock speedups."
    )


if __name__ == "__main__":
    main()

Execute it:

python speculative_demo.py

For the seeded example, the output is:

Largest 3-token outcome frequency gap: 0.0046
Baseline target steps per 64 tokens: 64
Illustrative verification passes per 64 tokens: 14.12
Draft acceptance fraction: 91.6%
These are algorithmic counts, not wall-clock speedups.

Interpreting the experiment

The first result compares empirical frequencies of complete three-event sequences.

The distributions are not expected to match perfectly after only 20,000 samples. A small difference is normal sampling noise.

The exactness guarantee is mathematical, not an assertion that two finite Monte Carlo experiments will produce identical histograms.

The second result is more interesting.

Ordinary generation needs 64 conceptual target decoding steps to produce 64 tokens.

The speculative implementation averages roughly 14 verification passes for the same output length.

That is a dramatic reduction in sequential target invocations.

However, 64 / 14.12 is not a measured inference speedup.

The demonstration ignores GPU kernel costs, draft-model latency, memory bandwidth, and the additional computation required to verify multiple tokens.

A real system must account for all of these.

The experiment establishes that the algorithm works and illustrates where its potential speedup originates.

6. Understand the mathematics of speculative speedups

Speculative decoding performance depends on three primary variables:

• Draft acceptance rate.

• Number of proposed tokens.

• Relative costs of drafting and verification.

Expected tokens per verification

Assume, for a simplified model, that each proposed token has the same independent conditional acceptance probability α.

If we propose k tokens, the expected number of emitted tokens, including a possible bonus token, is:

E[tokens] = 1 + α + α² + … + αᵏ

For example, consider four proposed tokens.

Expected tokens emitted per verification, k = 4:

This is the theoretical expected output count, assuming a uniform independent acceptance probability. The maximum is five tokens because of the bonus token.

Notice how much the acceptance rate matters.

A stronger draft predictor can be more valuable than simply increasing the speculation depth.

However, a stronger predictor may also be more expensive to execute.

That creates a trade-off.

A practical latency model

Let:

• T_target represent the cost of one ordinary target decode step.

• T_draft represent the cost of one draft step.

• T_verify(k) represent the cost of verifying k proposed tokens and obtaining the required next-token distribution.

• E represent the expected number of emitted tokens.

Then a simplified model is:

Baseline time ≈ E × T_target

Speculative time ≈ k × T_draft + T_verify(k)

Therefore:

Speedup ≈ (E × T_target) / (k × T_draft + T_verify(k))

This formulation reveals a surprising fact.

A draft model with better predictive accuracy is not automatically the best draft model.

For example, a larger draft model may increase acceptance from 80% to 90%, but if its inference cost doubles, total latency could become worse.

Optimizing speculation requires maximizing useful accepted output per unit of execution time, not merely maximizing accuracy.

Worked break-even examples

Plugging a few settings into the formula above, with every cost expressed as a percentage of one ordinary target decoding step (T_target = 100%):

The verification costs here are illustrative values, not hardware-derived predictions. Real verification cost changes with draft depth, model and batch size. Note how a cheap, accurate draft gives more than 2×, while a draft that is right only half the time gains only about 1.2×. Doubling the draft depth from 4 to 8 tokens adds nothing here, because the extra verification cost eats the extra accepted tokens.

The key engineering implication is that a good speculation configuration depends on the entire inference pipeline.

There is no universally optimal number of draft tokens.

7. Modern speculative decoding techniques

The classic small-draft-model approach is only one member of a much larger family of techniques.

Current inference systems offer several alternatives.

7.1 Standalone draft models

This is the most intuitive configuration.

A smaller transformer predicts tokens that are verified by a larger target transformer.

Advantages include:

• Relatively straightforward architecture.

• Compatibility with many target models.

• Independently trainable draft models.

• Potentially strong acceptance when the models are closely aligned.

The main drawbacks are extra memory usage and the cost of executing a second model.

The draft and target must also have compatible token semantics, or the runtime needs a correct cross-tokenizer verification mechanism.

7.2 EAGLE-3: feature-informed drafting

EAGLE-3 takes a more specialized approach.

Rather than relying only on a conventional smaller language model, it uses a lightweight drafting network informed by internal representations from the target model.

Its architecture combines information from multiple transformer layers and uses direct token prediction, improving on earlier EAGLE designs.

This provides richer information for predicting future tokens while keeping the draft network relatively inexpensive.

EAGLE-3 is especially interesting because it addresses the relationship between draft quality, training scale, and inference performance.

The vLLM and SGLang ecosystems both support EAGLE-family approaches, although actual model compatibility depends on the target architecture and available draft checkpoints.

7.3 Multi-token prediction

Some models have auxiliary prediction heads that can predict multiple future tokens.

Instead of launching an entirely independent draft transformer, the inference engine can use the model’s native multi-token prediction capabilities.

Advantages:

• Potentially less separate-model overhead.

• Tighter architectural integration.

• Useful acceleration when suitable auxiliary heads are available.

Limitations:

• Architecture-specific support.

• Checkpoint compatibility requirements.

• Different implementation details across serving frameworks.

In practice, multi-token prediction can be attractive when the original model was designed with this functionality.

7.4 N-gram and suffix-based speculation

Not every application needs a trained draft model.

Consider generating a long configuration file, structured JSON, or code containing repeated identifiers.

Large parts of the output may resemble tokens already present in the prompt or generation history.

N-gram speculation exploits this repetition by finding matching token sequences and proposing likely continuations.

Suffix-based methods similarly reuse token patterns, often through more sophisticated indexing structures.

These approaches can offer:

• No additional neural draft model.

• Lower memory overhead.

• Good performance for repetitive workloads.

• Simpler experimentation.

They are less effective when the generated text contains novel, difficult-to-predict continuations.

That makes them particularly workload-dependent.

Comparison of approaches

The choice depends on the target model, available GPU resources, output distribution, and expected concurrency.

8. The cherry on the cake: a reported 6.5× speedup

One of the most striking results comes from the EAGLE-3 research published at NeurIPS 2025.

The paper, EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, reports:

The distinction between these numbers is essential.

A maximum decoding speedup in a favorable benchmark does not imply that every production deployment will become 6.5 times faster.

At larger batch sizes, the GPU already processes substantial work concurrently, and the relative savings from speculation can shrink.

The broader lesson is more important than the headline number: speculative decoding can produce enormous gains in latency-sensitive regimes, while throughput improvements under heavy concurrency may be considerably smaller.

A newer finding from 2026

The ICML 2026 paper SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding studied how benchmarking methodology affects reported performance.

The researchers highlighted several practical issues:

• Synthetic inputs can overestimate real-world throughput.

• Optimal draft lengths depend on batch size.

• Low-diversity datasets can bias performance conclusions.

• Vocabulary pruning introduces additional evaluation caveats.

The paper integrates realistic serving-engine evaluations to make speculative decoding comparisons more representative of deployment conditions.

Another 2026 result, SpecGen, explores sparse self-speculation and reports up to 2.13× throughput acceleration across its evaluated settings. This shows that the field continues to move beyond the original separate draft-model design.

The takeaway is that a speedup number without workload, hardware, concurrency, and measurement context is incomplete.

9. Run speculative decoding with vLLM

Let’s move from the educational simulator to an actual inference server.

We will use vLLM’s n-gram speculation because it does not require downloading a separate trained draft model.

The example uses a Qwen3 instruction-tuned checkpoint and assumes a supported Linux GPU environment with sufficient memory.

Step 1: Install vLLM

Follow the current GPU installation instructions for the CUDA or other supported accelerator stack on your machine.

In a compatible environment, the standard Python installation command is:

python -m pip install vllm

Production installations should pin and test a supported version rather than upgrading dependencies blindly.

Step 2: Start a baseline server

First, launch the model without speculative decoding:

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
  --port 8000 \
  --max-model-len 4096

This establishes the comparison baseline.

Use the same model checkpoint, precision, hardware, model context limit, and sampling settings for the speculative run.

Step 3: Benchmark the baseline

In another terminal, run:

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --dataset-name random \
  --random-input-len 256 \
  --random-output-len 128 \
  --num-prompts 100 \
  --request-rate 1 \
  --seed 42

This is a useful first smoke test.

It measures a synthetic workload using a controlled request rate, prompt length, and output length.

Do not treat it as a production-representative benchmark, particularly for n-gram speculation. Random token sequences generally do not reproduce the repetition patterns of realistic application prompts.

Record:

• Output throughput.

• Mean and percentile time to first token.

• Inter-token latency.

• End-to-end request latency.

• GPU memory usage.

Step 4: Enable n-gram speculation

Stop the baseline server, then restart with speculative decoding enabled.

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
  --port 8000 \
  --max-model-len 4096 \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

The configuration defines a speculation strategy with a maximum of four proposed tokens per step.

The lookup settings control the token-pattern matching window.

These arguments follow the current vLLM speculative decoding configuration interface.

Step 5: Re-run the benchmark

Repeat the same benchmark command:

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --dataset-name random \
  --random-input-len 256 \
  --random-output-len 128 \
  --num-prompts 100 \
  --request-rate 1 \
  --seed 42

Compare results only after the server has completed model initialization and warmup.

Expect little or no gain from this particular run. --dataset-name random sends random token sequences, and n-gram speculation works by finding the continuation already written earlier in the prompt or output. Random text contains almost no such repeats, so most proposals are rejected. Treat this run as a check that the server works with speculation enabled. N-gram speculation pays off on repetitive workloads: code editing, summarizing or quoting a long document, and structured or templated output. SPEED-Bench (section 8) makes the same point about synthetic benchmark inputs.

For meaningful performance conclusions, repeat the experiment with representative application requests.

A useful real-world dataset might contain synthetic e-commerce support interactions, software configuration prompts, or generated server-log analysis requests.

Vary both request rate and output length.

A configuration that wins at one request per second may lose at higher concurrency.

Choosing speculation depth

An initial tuning sweep might try:

• Two draft tokens for conservative speculation.

• Four draft tokens as a balanced starting point.

• Six or eight tokens for highly predictable continuations.

These are experimental starting points, not universal recommendations.

The optimal value depends on acceptance behavior and the additional verification cost.

For richer model-based speculation, consult the current vLLM speculative decoding documentation and the SGLang speculative decoding guide.

10. Other optimizations that complement speculative decoding

Speculative decoding changes how token generation is scheduled.

Several other techniques improve the cost of executing the transformer itself.

These approaches often complement speculation, but their benefits are not necessarily additive.

10.1 Quantization: move fewer bytes

Quantization reduces the precision used to represent model parameters.

Typical deployment options include:

• BF16 or FP16 for conventional half-precision weights.

• FP8 for supported weights, activations, or caches.

• INT8 for compatible quantized execution.

• INT4 or other low-bit formats for aggressive memory reduction.

For a 14-billion-parameter model, approximate raw weight storage is:

These are theoretical weight-storage figures, not complete deployment memory requirements.

Real quantized checkpoints also contain scaling metadata and may require workspace buffers, non-quantized tensors, and dequantization operations.

Quantization can accelerate memory-bound inference by reducing bytes transferred.

However, it does not guarantee faster execution.

Low-bit formats need efficient hardware kernels, and some schemes introduce enough conversion overhead to offset their memory savings.

Quantization can also slightly change model outputs, unlike correctly implemented lossless speculative sampling against an unchanged target.

10.2 KV-cache quantization

The KV cache grows with the number of active sequences and their retained context lengths.

For conventional full-attention transformer architectures, its storage depends on:

• Layer count.

• Number of KV heads.

• Head dimension.

• Retained token count.

• Element precision.

• Number of concurrent sequences.

Moving KV-cache storage from 16-bit to 8-bit precision can approximately halve the raw cache element footprint.

This may enable larger batches or longer contexts within the same GPU memory budget.

Modern inference frameworks support quantized KV-cache formats, including FP8 on compatible hardware.

The trade-off involves conversion overhead, hardware support, and potential numerical effects.

10.3 Prefix caching

Many applications repeatedly send the same instructions or reference material.

Imagine a retail assistant receiving hundreds of requests that begin with:

You are a customer support assistant. Follow these store policies…

If each request contains the same long prefix, recomputing all of its attention states wastes resources.

Automatic prefix caching allows the runtime to reuse previously computed KV-cache blocks for matching token prefixes.

This can reduce prefill work significantly.

It is especially valuable for:

• Shared system prompts.

• Repeated documentation context.

• Multi-turn conversations.

• Applications with large reusable instructions.

Prefix caching does not directly eliminate the sequential generation bottleneck. Its main benefit is avoiding redundant prompt processing.

vLLM documents automatic prefix caching as a mechanism for reusing KV state across requests with matching prefixes.

10.4 Continuous batching

Traditional static batching groups requests and processes them together.

The problem is that requests have different output lengths.

A short request may finish while a longer request continues generating, wasting batch capacity if the system cannot efficiently replace completed work.

Continuous batching allows a serving engine to add and remove requests as decoding progresses.

This improves utilization by keeping the GPU occupied with useful work.

The main benefit is often increased serving throughput rather than reduced standalone single-request latency.

Its interaction with speculative decoding is important.

At high concurrency, executing many normal decode operations together may already make excellent use of GPU compute. Speculative verification may then produce smaller relative gains.

10.5 Chunked prefill

Large prompts can create scheduling problems.

If a long prefill monopolizes the GPU, existing users may experience delays between generated tokens.

Chunked prefill splits long prompt processing into smaller segments that can be scheduled alongside decoding operations.

This helps balance the competing needs of:

• New requests entering the system.

• Existing requests waiting for their next token.

• GPU utilization.

• Time-to-first-token latency.

In current vLLM V1 configurations, chunked prefill is enabled by default when supported. The scheduler can prioritize decoding and fit prefill chunks into the remaining token budget.

10.6 Efficient attention kernels

Attention kernels determine how efficiently the model reads and combines previous token representations.

Modern implementations use techniques such as:

• Fused GPU operations.

• Tiled attention computation.

• Reduced intermediate-memory traffic.

• Efficient causal masking.

• Hardware-specific matrix instructions.

FlashAttention-style kernels are particularly important during prefill and attention-heavy workloads.

They reduce memory movement relative to materializing large attention matrices.

At long context lengths, efficient attention can also matter during decoding.

The best kernel depends on architecture, sequence length, hardware, KV-cache format, and batch shape.

10.7 CUDA graphs and kernel fusion

For small decoding workloads, GPU execution overhead can become significant.

Even when kernels execute quickly, launching many small operations from the host incurs overhead.

CUDA graphs allow compatible GPU operation sequences to be captured and replayed with reduced launch overhead.

Kernel fusion combines multiple operations into fewer launches and may reduce intermediate memory movement.

These techniques can improve token latency, but come with trade-offs:

• Additional graph-capture memory.

• Constraints around dynamic shapes.

• Compilation and warmup overhead.

• More complicated scheduling.

They are useful complements to speculative decoding, particularly when drafting introduces many small GPU operations.

11. How the optimizations interact

A common mistake is to multiply independently reported speedups.

Suppose one benchmark reports:

• 2× improvement from speculation.

• 1.4× from quantization.

• 1.2× from another kernel optimization.

It is tempting to multiply them and claim an overall 3.36× improvement.

That is not generally valid.

Each optimization changes the underlying system bottlenecks.

For example, quantization can make target-model decoding faster, reducing the relative benefit of speculative decoding.

Likewise, increasing concurrency can improve GPU utilization while making additional speculation less attractive.

The correct approach is to benchmark the combined configuration.

A powerful production configuration usually combines several optimizations, but only after identifying which bottleneck dominates.

12. Benchmark the metrics that actually matter

An inference server should not be evaluated using tokens per second alone.

Different applications have very different performance requirements.

Time to first token

Time to first token, or TTFT, measures the delay before the first generated token becomes available.

It includes factors such as queueing, prompt processing, and the initial generation work.

It is critical for interactive applications.

Speculative decoding does not automatically reduce TTFT. In some deployments, initializing draft state or coordinating verification may add work before useful output appears.

Prefix caching and prefill scheduling may be more relevant optimizations for this metric.

Inter-token latency

Inter-token latency measures the delay between output tokens.

This is where speculative decoding often provides the clearest benefit.

However, streaming behavior introduces a complication.

A verifier may produce multiple accepted tokens in a burst, followed by a longer wait for the next verification.

Therefore, an average inter-token metric may not fully describe the user’s perceived streaming smoothness.

Inspect latency distributions and output timing, not only arithmetic means.

End-to-end latency

End-to-end latency measures the duration from request submission to completion.

For long responses, improving decode efficiency can significantly reduce this metric.

For short responses with long prompts, prefill may dominate, limiting the benefit of speculation.

Throughput

Useful throughput can be measured as:

Output throughput = completed output tokens / elapsed time

For multi-user systems, also track completed requests per second.

These metrics should be measured alongside latency rather than replacing it.

A server that maximizes tokens per second but produces unacceptable user-facing delays may not be a good deployment.

Tail latency

The 95th and 99th percentile latencies matter because production workloads are heterogeneous.

Some requests have long prompts, some generate long responses, and some wait behind expensive operations.

A speculative configuration that improves average latency while worsening P99 may be undesirable for latency-sensitive services.

Draft acceptance statistics

For speculative decoding specifically, track:

• Total proposed tokens.

• Accepted draft tokens.

• Acceptance fraction.

• Average accepted prefix length.

• Mean emitted tokens per verification.

• Target verification time.

• Draft execution time.

• Rejection frequency by draft position.

Acceptance fraction and average accepted length are different metrics.

For example, a draft strategy may have a respectable aggregate acceptance fraction while frequently rejecting its first or second token, limiting useful progress per verification.

Modern vLLM documentation includes per-request acceptance metrics and facilities for evaluating speculative behavior.

13. When speculative decoding can make things worse

It is tempting to enable speculation everywhere.

That is a mistake.

Case 1: An expensive draft model

If the draft model consumes too much computation, it can erase the savings from reducing target-model invocations.

A small model is not necessarily cheap enough.

GPU kernel efficiency, memory placement, and synchronization are often just as important as parameter count.

Case 2: Low draft acceptance

Consider a task requiring highly varied creative writing.

The draft and target distributions may diverge substantially.

A large fraction of speculative tokens can be rejected, meaning the system pays drafting and verification overhead without committing many additional tokens.

For these workloads, shorter speculation lengths may perform better.

Case 3: Large inference batches

When the target model processes many requests concurrently, its matrix operations can become more compute-efficient.

The verifier may no longer have the same advantage over standard decoding as it does at batch size one.

This is one reason batch-aware speculation is an active research and engineering area.

Case 4: Short outputs

Suppose an application generates only five tokens per request.

Speculation has very little sequential work to eliminate.

Additional draft coordination could make the system slower.

For these workloads, scheduling and prefill optimizations may matter more.

Case 5: Memory pressure

A separate draft model needs memory for weights, state, and runtime buffers.

On a GPU already close to capacity, enabling speculation can reduce available KV-cache memory and therefore decrease the maximum useful concurrency.

A faster individual decoding path may then produce lower total system throughput.

Case 6: Incompatible decoding rules

Correct verification must account for the relevant sampling transformations.

Temperature, top-p filtering, constrained output formats, token masks, and other logits transformations must be handled consistently.

If the target probabilities used by the verifier do not correspond to the intended target distribution, the theoretical preservation guarantee no longer follows automatically.

Production implementations should be tested across all decoding modes the application actually uses.

14. A practical optimization workflow

Rather than enabling every feature simultaneously, use a controlled sequence of experiments.

Phase A: Establish a baseline

Measure the current deployment under a representative workload.

Record:

• Hardware and runtime versions.

• Model and precision.

• Request arrival rate.

• Input and output length distributions.

• Batch size or concurrency.

• TTFT, inter-token latency, and throughput.

• GPU memory utilization.

Without this baseline, optimization results are difficult to interpret.

Phase B: Identify the dominant bottleneck

Profile the runtime.

If the problem is long prompt processing, prioritize efficient attention, prefix caching, and prefill scheduling.

If the problem is sequential low-batch decoding, evaluate speculation.

If the problem is insufficient GPU utilization, prioritize batching and request scheduling.

If the problem is memory capacity, evaluate quantization and KV-cache management.

Phase C: Add one optimization at a time

A reasonable sequence for a memory-bound decoding workload is:

1. Confirm the serving engine uses efficient kernels.

2. Tune memory allocation and KV-cache capacity.

3. Evaluate supported lower-precision execution.

4. Establish latency and throughput measurements.

5. Enable an inexpensive speculation method.

6. Tune the speculation depth.

7. Evaluate stronger drafting approaches such as EAGLE-3.

8. Test the winning configurations across different concurrency levels.

Do not assume that every optimization will improve every workload.

Phase D: Validate correctness

Performance is only useful if the model still behaves as intended.

Check:

• Output quality on representative prompts.

• Structured-output validity.

• Stop sequence and end-of-sequence handling.

• Sampling behavior.

• Token-length constraints.

• Compatibility with application-specific logits processing.

• Memory stability over sustained operation.

For exact speculative decoding, statistical tests can help detect sampling inconsistencies, although finite-sample tests cannot prove distributional equivalence.

Phase E: Deploy behind a measurable rollout

Production changes should be observable and reversible.

Track latency, throughput, GPU memory, error rates, and acceptance behavior.

Consider maintaining separate configurations for latency-sensitive and high-throughput traffic.

A low-concurrency coding assistant and a heavily batched document-processing service may benefit from entirely different inference strategies.

15. What comes next for fast inference?

Speculative decoding is evolving from a relatively simple two-model algorithm into a family of hardware- and workload-aware generation techniques.

Several directions are particularly promising.

Adaptive draft depth allows the runtime to propose more tokens when confidence is high and fewer when the predicted continuation is uncertain.

Tree-based speculation proposes several possible continuations rather than following one linear draft path. The verifier can evaluate an appropriately masked candidate tree and accept a valid continuation.

Parallel drafting tries to reduce the sequential work performed by the draft predictor itself.

Self-speculation reuses the target model’s architecture or parameters, potentially through early exits, selective computation, or sparse attention.

Scheduler-aware speculation adapts drafting decisions to system load. When batch pressure is high, the server can reduce speculative work and prioritize throughput.

Hardware-aware verification optimizes memory movement, attention kernels, and execution layout specifically for candidate blocks rather than treating verification as ordinary prefill.

These developments point toward a broader principle:

The most efficient decoding system may no longer be the one that performs the cheapest possible single-token forward pass.

Instead, it may be the one that coordinates prediction, verification, caching, and scheduling to maximize useful output per unit of time and hardware cost.

16. Key takeaways

Speculative decoding addresses one of the most expensive characteristics of autoregressive generation: repeatedly executing a large target model to produce individual tokens.

The most important lessons are:

• Speculation trades cheap predictions for fewer sequential expensive target passes. Its value depends on the relationship between drafting cost, verification cost, and accepted tokens.

• Exact speculative sampling preserves the target distribution. Correct acceptance ratios, residual sampling, and rejection handling are essential.

• Acceptance rate alone does not determine performance. A slightly weaker but much cheaper draft model may deliver better latency.

• Different workloads need different speculation strategies. EAGLE-3, native multi-token prediction, n-gram lookup, and self-speculation have different strengths.

• Reported speedups are context-dependent. Research has demonstrated substantial gains, including an EAGLE-3 result of up to 6.5×, but production throughput improvements can be considerably smaller.

• The fastest serving stacks combine techniques intelligently. Quantization, KV-cache optimization, prefix reuse, scheduling, and efficient kernels all affect the final result.

The central engineering lesson is simple: optimize the full request lifecycle, not just one transformer operation.

17. References and further reading

The following sources provide research findings and current implementation guidance:

• Li et al., EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. Paper.

• Abramovich et al., SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding. ICML 2026. Paper.

• Zhao et al., Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding. MLSys 2026. Paper.

• vLLM documentation. Speculative decoding methods and configuration.

• vLLM documentation. Inference optimization and tuning.

• SGLang. Open-source inference framework.

18. Your next step: measure a real speedup

Start with the Python implementation from this tutorial. Experiment with changing the draft distributions and observe how proposal acceptance, expected output length, and verification counts change.

Then choose one real model and one representative application workload.

Benchmark the same serving configuration with speculative decoding disabled and enabled, using several draft depths and concurrency levels.

Your first objective should not be to reproduce a headline 6× research result.

Aim for a reproducible 2× improvement in the metric your users actually experience, without sacrificing correctness or reliability.

If you can achieve that consistently, you have moved beyond an interesting inference experiment and built a meaningful production optimization.