LEARN · MULTI-MODEL INFERENCE & SERVING
The serving bottleneck is not just FLOPs
A language model can generate only one next token per sequence at a time. That sounds like an inherently serial workload, and at the level of one request it largely is.
Serving many users changes the picture.
A production inference server may have dozens or hundreds of sequences at different stages:
-
one request is processing a 2,000-token prompt;
-
another is generating token 47;
-
another just finished;
-
three new requests have arrived;
-
several long-running generations need their accumulated attention state kept in GPU memory.
The GPU would like to process these requests together. Larger batches amortize kernel-launch overhead, improve utilization, and turn a stream of tiny decode operations into substantial parallel work.
The obstacle is memory, especially the key-value cache, or KV cache.
Modern vLLM still describes efficient KV-cache management with PagedAttention as a core part of its serving design, but the current engine combines that foundation with continuous batching, chunked prefill, optimized attention kernels, CUDA/HIP graphs, prefix caching, speculative decoding, and other execution optimizations. In other words, it would be misleading to attribute every modern vLLM speedup to one algorithm.
The important causal chain is:
better KV-cache memory utilization → more live sequences fit on the GPU → larger effective batches → better GPU utilization → much higher aggregate throughput.
That is why a memory-management idea could produce an order-of-magnitude serving improvement without changing the model’s weights.
First understand what the KV cache is storing
During autoregressive generation, a transformer repeatedly computes attention over the existing context.
Conceptually, attention is:
Attention(Q, K, V) = softmax(Q·Kᵀ / √d_head) · V
When the model generates the next token, almost all previous keys and values are unchanged. Recomputing them from the entire prefix at every decoding step would be enormously wasteful.
Instead, inference engines keep the old keys and values in a KV cache. Hugging Face’s current Transformers documentation describes the same mechanism: previous K and V tensors are retained and reused while the cache grows as new tokens are produced.
For a conventional full-attention layer, a useful approximation is:
KV bytes per token = number of layers × 2 × number of KV heads × head dimension × bytes per element
The factor of 2 exists because there is one cache for keys and another for values.
Consider a hypothetical decoder with:
-
32 layers;
-
8 KV heads;
-
head dimension 128;
-
BF16 cache entries, at 2 bytes each.
Its KV-cache footprint per token is:
32 × 2 × 8 × 128 × 2 = 131,072 bytes
That is 128 KiB per token.
An 8,192-token sequence therefore consumes roughly:
8,192 × 128 KiB = 1 GiB
That is for one sequence.
The exact number varies dramatically with architecture. Grouped-query attention reduces the number of KV heads, hybrid architectures can use different cache types across layers, and current vLLM even exposes multiple KV-cache data types and specialized cache handling. But the core systems problem remains: cache capacity grows with active context, while request lengths are unpredictable.
Why naive allocation wastes so much memory
Imagine implementing an inference server with the simplest possible allocator.
A request arrives and may eventually grow to 4,096 tokens. You reserve one contiguous KV-cache region large enough for 4,096 tokens.
Then the request finishes after 287 tokens.
Most of the reservation was unnecessary.
A second problem appears when allocations have different lifetimes. Suppose GPU memory looks conceptually like this:
[ request A ][ request B ][ request C ][ request D ]
Request B finishes:
[ request A ][ free ][ request C ][ request D ]
A new request needs more memory than the hole occupied by B. There may be enough free memory in total, but not enough contiguous memory in the place where the allocator wants it.
That is fragmentation.
The third problem is that generation length cannot normally be known in advance. If you allocate conservatively, you limit concurrency. If you reserve each request’s theoretical maximum, you over-allocate.
Traditional tensor layouts are much happier when memory is regular and contiguous. Online LLM workloads are anything but regular.
That mismatch was the key insight behind PagedAttention.
PagedAttention borrows an operating-system trick
An operating system does not normally require every process to occupy one enormous physically contiguous region of RAM corresponding exactly to its virtual address space.
It divides memory into pages and maintains mappings from a process’s virtual addresses to physical locations.
PagedAttention applies a closely related abstraction to KV-cache storage.
Instead of demanding one contiguous physical KV allocation for every sequence, vLLM divides a sequence’s cache into fixed-size blocks. A logical sequence of KV blocks can then map to physical blocks that are scattered around the available cache memory.
Conceptually:
Logical KV blocks for request A: A0 -> A1 -> A2 -> A3 Physical cache: slot 0: B1 slot 1: A3 slot 2: free slot 3: A0 slot 4: C2 slot 5: A2 slot 6: A1
The logical sequence remains ordered. The physical memory does not have to be.
The attention implementation knows how to follow the block mapping when it needs the keys and values.
Current vLLM documentation still describes its paged KV-cache representation in terms of blocks and block tables, while the project’s current feature set builds many newer optimizations around that memory-management foundation.
This changes the allocation problem fundamentally.
Instead of asking:
Can I find one large contiguous region for this sequence?
the memory manager can ask:
How many free blocks does this sequence need right now?
A request grows by allocating additional blocks as necessary.
When the request ends, its blocks return to the pool and can immediately be reused by unrelated requests.
The surprising result: memory efficiency becomes compute throughput
It is tempting to describe PagedAttention as a memory optimization.
That is true but incomplete.
Suppose a naive serving design can keep only eight active sequences resident because its KV allocations waste significant GPU memory.
If better cache management lets the same GPU keep 40 sequences active, the scheduler suddenly has many more requests from which to construct useful batches.
The model did not get mathematically cheaper.
The GPU has simply stopped spending so much of its serving capacity on unusable reservations and fragmented cache space.
This distinction explains why benchmark headlines can seem astonishing. A 10× throughput result does not mean one matrix multiplication became ten times faster.
Instead, the whole system can perform far more useful work per second under concurrency.
That is also why the improvement is workload-dependent.
If you benchmark a single request at concurrency one, there is little opportunity for a serving engine to exploit high-throughput scheduling. Your latency may be dominated by the same model kernels regardless of memory allocator.
Increase concurrent requests, vary their lengths, and put pressure on the KV cache, and the architecture starts to matter much more.
Continuous batching is the other half of the story
Static batching is straightforward:
-
collect N requests;
-
pad them as necessary;
-
process the batch;
-
keep every member in that batch until the whole operation is complete.
That works poorly for online generation because requests finish at different times.
If one sequence needs 20 output tokens and another needs 500, tying them together wastes scheduling opportunities.
A continuous-batching scheduler operates at a finer granularity. Finished requests leave, and waiting requests can enter execution without waiting for an entire static batch lifecycle to end.
Current vLLM goes further. In its V1 engine, chunked prefill is enabled by default whenever possible. The scheduler prioritizes pending decode work, then uses remaining token budget for prefills; large prefills can be split into chunks. vLLM’s current tuning documentation says this can combine memory-bound decode work with compute-bound prefill work, improving GPU utilization while balancing throughput, time to first token, and inter-token latency.
That matters because prefill and decode stress hardware differently.
During prefill, the model processes many prompt tokens together. Matrix multiplications are large enough that the phase tends to be relatively compute-heavy.
During decode, each sequence contributes only its next token while repeatedly reading model weights and cache state. Decode is often more constrained by memory movement.
A good server tries to construct work that keeps the accelerator productive across both regimes.
Paged KV management gives the scheduler flexibility. Continuous batching decides how to use it.
A current, runnable experiment
Rather than accepting a historical “10×” number, you should reproduce the shape of the effect on your own GPU.
The experiment below compares:
-
a deliberately naive Hugging Face Transformers loop processing one request at a time;
-
a current vLLM OpenAI-compatible server;
-
several concurrency levels on the vLLM side.
The baseline still uses a KV cache. We are not sabotaging Transformers by recomputing the whole prefix for every token. “Naive” here means sequential generate() calls without a dedicated continuous-batching serving engine.
For the model, we will use Qwen/Qwen3-0.6B. That is also the default model shown by the current vLLM serve CLI reference, making it a convenient small checkpoint for an infrastructure experiment.
The model is intentionally small. The point is to inspect serving behavior, not to claim that this exact checkpoint will produce the largest possible vLLM advantage.
Hardware assumptions
For the commands below, assume:
-
Linux;
-
an NVIDIA GPU;
-
a BF16-capable GPU;
-
enough VRAM for the model plus its runtime state;
-
Python in the range supported by current vLLM releases.
Current vLLM documentation lists Python 3.10 through 3.13 for its quickstart and supports direct pip installation for NVIDIA systems.
Create an isolated environment:
python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install "vllm[bench]"
The benchmark extra is intentional: current vLLM documentation specifies vllm[bench] for the benchmark CLI.
Before benchmarking, record what you actually ran:
python -c "import torch, transformers, vllm; print('torch', torch.__version__); print('transformers', transformers.__version__); print('vllm', vllm.__version__)" nvidia-smi
Do not skip this step when publishing results. “I tested vLLM” is insufficient information in a fast-moving inference stack.
Baseline: sequential Hugging Face generation
Create hf_naive.py:
import argparse import json import time import torch from transformers import AutoModelForCausalLM def parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--model", default="Qwen/Qwen3-0.6B") parser.add_argument("--num-prompts", type=int, default=256) parser.add_argument("--input-len", type=int, default=1024) parser.add_argument("--output-len", type=int, default=128) parser.add_argument("--seed", type=int, default=0) return parser.parse_args() def main(): args = parse_args() if not torch.cuda.is_available(): raise RuntimeError("This benchmark requires a CUDA GPU.") torch.manual_seed(args.seed) torch.cuda.manual_seed_all(args.seed) model = AutoModelForCausalLM.from_pretrained( args.model, dtype=torch.bfloat16, ).to("cuda") model.eval() vocab_size = model.config.vocab_size prompts = torch.randint( low=0, high=vocab_size, size=(args.num_prompts, args.input_len), dtype=torch.long, device="cuda", ) warmup_ids = prompts[0:1] warmup_mask = torch.ones_like(warmup_ids) with torch.inference_mode(): model.generate( input_ids=warmup_ids, attention_mask=warmup_mask, do_sample=False, min_new_tokens=16, max_new_tokens=16, use_cache=True, ) torch.cuda.synchronize() start = time.perf_counter() generated_tokens = 0 with torch.inference_mode(): for i in range(args.num_prompts): input_ids = prompts[i : i + 1] attention_mask = torch.ones_like(input_ids) output_ids = model.generate( input_ids=input_ids, attention_mask=attention_mask, do_sample=False, min_new_tokens=args.output_len, max_new_tokens=args.output_len, use_cache=True, ) generated_tokens += output_ids.shape[1] - args.input_len torch.cuda.synchronize() elapsed = time.perf_counter() - start result = { "backend": "hf-sequential", "model": args.model, "num_prompts": args.num_prompts, "input_len": args.input_len, "output_len": args.output_len, "elapsed_s": round(elapsed, 3), "requests_per_s": args.num_prompts / elapsed, "output_tokens_per_s": generated_tokens / elapsed, "total_tokens_per_s": ( args.num_prompts * args.input_len + generated_tokens ) / elapsed, } print(json.dumps(result, indent=2)) if __name__ == "__main__": main()
Run it:
python hf_naive.py \ --model Qwen/Qwen3-0.6B \ --num-prompts 256 \ --input-len 1024 \ --output-len 128
The script deliberately pre-generates the synthetic token IDs before timing. That keeps random-number generation and host-to-device prompt transfer out of the measurement.
It therefore gives the sequential baseline a slight methodological advantage over an HTTP server benchmark, which still has front-end and request-management work to do.
That is useful. If the serving engine wins convincingly despite carrying more serving machinery, the result is more interesting.
The current Transformers API continues to expose AutoModelForCausalLM and generate() for causal generation, and its documentation explicitly describes KV caching as part of efficient autoregressive inference.
Start the vLLM server
Use a separate terminal in the same environment:
source .venv/bin/activate vllm serve Qwen/Qwen3-0.6B \ --host 127.0.0.1 \ --port 8000 \ --dtype bfloat16 \ --max-model-len 4096
The current serve CLI supports bfloat16 explicitly and derives model length automatically when you do not override it. Here we intentionally cap the maximum model length because our workload requires only 1,024 input tokens plus 128 generated tokens.
Reducing an unnecessarily huge context limit is not a universal performance trick, but explicit benchmark limits make experiments easier to reason about.
Also notice what we did not add:
-
no speculative decoding;
-
no quantization;
-
no prefix-reuse workload;
-
no multi-GPU tensor parallelism;
-
no hand-tuned batch-token limit.
We want a clean serving comparison before stacking extra optimizations on top.
Benchmark the server at concurrency one
Start with the least favorable environment for high-throughput serving:
vllm bench serve \ --backend vllm \ --model Qwen/Qwen3-0.6B \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 128 \ --num-prompts 256 \ --max-concurrency 1 \ --ignore-eos \ --num-warmups 8 \ --save-result \ --result-filename vllm-c1.json
The current benchmark CLI supports a synthetic random dataset, fixed input and output lengths, explicit maximum concurrency, warmups, and --ignore-eos.
--ignore-eos matters because we want each request to generate the requested number of tokens instead of letting different early-stop positions distort throughput.
At concurrency one, do not be surprised if the comparison looks unimpressive.
That is a feature of the experiment, not a failure.
Paged KV allocation cannot manufacture concurrency that does not exist.
Now sweep concurrency
Run exactly the same token lengths while allowing more requests to overlap:
for concurrency in 1 4 16 64; do vllm bench serve \ --backend vllm \ --model Qwen/Qwen3-0.6B \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 128 \ --num-prompts 256 \ --max-concurrency "${concurrency}" \ --ignore-eos \ --num-warmups 8 \ --save-result \ --result-filename "vllm-c${concurrency}.json" done
Because the benchmark’s default request rate is effectively unbounded, the client can offer requests as quickly as the concurrency cap allows. The current documentation distinguishes that request-arrival rate from --max-concurrency, which controls how many requests may actually be executing simultaneously.
Plot or tabulate at least:
-
requests per second;
-
output tokens per second;
-
total token throughput;
-
time to first token;
-
inter-token latency or time per output token;
-
peak GPU memory;
-
any server-side preemption warnings.
The most revealing graph is usually output tokens per second versus concurrency.
The sequential baseline gives you essentially one point.
The vLLM curve tells you how effectively the engine turns additional demand into useful GPU work.
What a convincing result should look like
Do not judge the experiment by asking only:
Is vLLM 10× faster?
Ask instead:
How does throughput scale as I provide the server with more concurrent work?
A typical qualitative shape is:
Throughput ^ | __________ | ____/ | ____/ | ____/ | ____/ |____/ +----------------------------------> concurrency
Initially, throughput rises rapidly because the server builds more effective batches.
Eventually it plateaus because another resource becomes saturated:
-
memory bandwidth;
-
tensor-core compute;
-
KV-cache capacity;
-
scheduler token budget;
-
model execution overhead;
-
GPU memory capacity.
If you push beyond the efficient region, latency can continue to deteriorate without a corresponding throughput gain.
That is why maximum throughput is not automatically the correct production configuration.
Throughput and latency are different objectives
Suppose configuration A produces 4,000 output tokens per second with excellent time to first token.
Configuration B produces 4,800 output tokens per second but doubles P99 latency.
B wins the throughput benchmark and may still be the wrong choice for an interactive assistant.
Production serving usually has several objectives:
-
maximize tokens per second;
-
keep time to first token within an SLO;
-
control inter-token latency;
-
avoid pathological tail latency;
-
leave enough cache headroom to absorb bursts;
-
avoid repeated preemption and recomputation.
Current vLLM can preempt requests when KV-cache space runs short. Its documentation warns that repeated preemption hurts end-to-end performance and suggests increasing available KV-cache memory or reducing scheduling pressure when it happens frequently.
So when your benchmark shows throughput collapsing at very high concurrency, inspect the logs before declaring the GPU “too slow.”
You may simply have driven the cache beyond its efficient operating point.
Why longer contexts amplify the memory problem
Repeat the benchmark with larger prompts:
for input_len in 128 1024 2048; do vllm bench serve \ --backend vllm \ --model Qwen/Qwen3-0.6B \ --dataset-name random \ --random-input-len "${input_len}" \ --random-output-len 128 \ --num-prompts 256 \ --max-concurrency 64 \ --ignore-eos \ --num-warmups 8 \ --save-result \ --result-filename "vllm-input-${input_len}.json" done
As contexts grow, each live request owns more cache state.
That means the engine faces a direct concurrency trade-off:
longer active contexts → more cache per request → fewer simultaneously resident sequences, all else equal.
The exact curve depends on architecture. Models using fewer KV heads can have dramatically lower per-token cache footprints than otherwise similar multi-head-attention models.
This is why “parameter count” alone is a poor predictor of serving capacity.
Two models with similar weight size may require very different KV-cache budgets.
A useful calculation before deploying any model
Before buying GPUs, inspect the architecture and estimate its cache footprint.
For a standard attention model:
KV bytes/token ≈ layers × 2 × KV heads × head dimension × cache element bytes
Then:
KV bytes/request ≈ KV bytes/token × expected active sequence length
Then:
approximate cache-limited concurrency ≈ available KV-cache bytes / KV bytes/request
This is only a first-order estimate. Real engines need block granularity, workspace memory, scheduler headroom, graph allocations, model-specific states, and potentially multiple cache groups.
But it immediately tells you whether your workload is fundamentally weight-dominated or KV-dominated.
For example, if each request consumes roughly 512 MiB of cache at your target context length and the engine has 16 GiB available for KV state, expecting hundreds of simultaneously resident long-context requests is unrealistic no matter how fast your GPU’s matrix multiplications are.
Why batching one giant tensor is not the same thing
At this point, you might ask: why not simply call Hugging Face generate() with a batch of 64 prompts?
You can, and for uniform offline jobs it may work well.
But that changes the workload.
Production traffic has:
-
requests arriving at different times;
-
different prompt lengths;
-
different output lengths;
-
cancellations;
-
streaming;
-
bursts;
-
long generations mixed with short ones.
A static batch assumes enough uniformity to construct the batch in advance.
An online inference engine must continuously reorganize work while preserving the KV state of every live sequence.
That dynamic systems problem is precisely where block-based cache management and continuous scheduling earn their keep.
For a fairer offline batch-throughput comparison, use batched Transformers and vLLM’s offline benchmark tooling. For a serving comparison, however, a sequential baseline is useful because it exposes what a general model library does not automatically provide: a dedicated concurrent request scheduler.
Why the headline speedup can shrink today
There is another reason you should not treat a historical 10× or 20× number as a timeless constant: competing stacks improve.
Modern Transformers itself has multiple cache implementations and an increasingly optimized generation stack. Hugging Face’s current cache documentation distinguishes dynamic caches, static caches, and sliding-window variants, with static layouts offering compilation-friendly behavior.
The comparison therefore depends on what you mean by “Hugging Face.”
A deliberately sequential generate() loop is not the same thing as:
-
a compiled static-cache workload;
-
an optimized batch-generation script;
-
a dedicated Transformers serving implementation;
-
another production inference engine.
This course uses naive generation because that is the requested baseline and because it cleanly demonstrates the systems value of serving-aware scheduling.
It should not be presented as evidence that every possible Transformers-based setup is an order of magnitude slower.
PagedAttention is not FlashAttention
The names are similar enough that people frequently conflate them.
They solve different problems.
FlashAttention is primarily about executing the attention computation efficiently, especially reducing unnecessary memory traffic and materialization inside the attention operation.
PagedAttention is primarily about how the persistent KV cache for active sequences can be organized and addressed efficiently in an online serving system.
A modern inference stack can use both ideas.
One optimizes attention execution.
The other enables flexible cache-memory management for many changing sequences.
That distinction is important because serving throughput is the product of multiple layers:
model architecture
|
attention/GEMM kernels
|
KV-cache representation
|
scheduler and batching policy
|
request/API layer
|
production traffic
Optimizing only one layer eventually exposes the next bottleneck.
Prefix caching becomes natural once cache blocks are first-class objects
Consider a service whose requests often begin with the same large prefix:
-
the same system prompt;
-
the same tool definitions;
-
the same document;
-
the same few-shot examples.
Without prefix caching, every request performs the same prefill work again.
vLLM’s current automatic prefix caching feature can reuse cached KV state when new requests share an existing prefix, allowing the engine to skip computation for that shared portion.
This is closely related to the block-management worldview.
Once cached state has explicit block identity and lifecycle management, blocks are not merely scratch space belonging to one monolithic request allocation. They can become reusable serving objects.
Prefix caching does not make decoding tokens intrinsically faster. It reduces repeated prefill work.
That distinction matters when interpreting a benchmark.
If you benchmark unique random prompts, you are largely measuring scheduling and generation behavior.
If every request shares a 10,000-token prefix, cache reuse can dominate the result.
Always report which workload you used.
The cherry on the cake: the breakthrough came from operating systems
The most interesting part of the PagedAttention story is that its central metaphor did not come from inventing a new transformer equation.
It came from a decades-old operating-systems abstraction.
The original vLLM work explicitly described PagedAttention as inspired by virtual memory and paging. The idea was to stop treating a sequence’s logical KV cache as though it had to correspond to one physically contiguous allocation.
That conceptual jump produced spectacular historical benchmark numbers.
In the original June 2023 vLLM launch experiments, the team compared LLaMA-7B on an NVIDIA A10G and LLaMA-13B on a 40 GB A100 using request-length distributions sampled from ShareGPT. They reported up to 24× the throughput of the Hugging Face Transformers baseline and up to 3.5× the throughput of the then-compared Hugging Face Text Generation Inference system. For the single-completion workload, the published chart reported a 14×–24× range against the Transformers baseline.
Those are historical results from 2023, not current promises.
That distinction is crucial.
The remarkable lesson is not “install package X and your model becomes 24× faster.”
It is that a supposedly low-level memory-allocation constraint was preventing the GPU from seeing enough concurrent useful work. Once the memory system changed, the entire serving throughput regime changed.
A classic OS idea unlocked a modern AI workload.
What current vLLM has become
PagedAttention may be the architectural idea that made vLLM famous, but current vLLM is much broader.
The project’s current documentation lists capabilities including:
-
continuous batching;
-
chunked prefill;
-
prefix caching;
-
CUDA and HIP graphs;
-
multiple optimized attention kernels;
-
quantization across numerous formats;
-
speculative decoding;
-
tensor, pipeline, data, expert, and context parallelism;
-
disaggregated prefill and decode;
-
OpenAI-compatible serving;
-
streaming;
-
structured output;
-
multi-LoRA serving.
This is why “PagedAttention benchmark” and “vLLM benchmark” are no longer synonyms.
A current vLLM result measures the integrated engine.
PagedAttention explains an important part of the architecture and history, but modern throughput comes from the whole serving stack.
How to run an empirical benchmark that is worth publishing
A credible benchmark should freeze more than the model name.
Record:
-
exact GPU model and VRAM;
-
GPU count;
-
driver version;
-
CUDA/runtime environment;
-
vLLM version;
-
PyTorch version;
-
Transformers version;
-
model repository and, ideally, revision;
-
model dtype;
-
KV-cache dtype;
-
tensor-parallel configuration;
-
maximum model length;
-
input-length distribution;
-
requested output-length distribution;
-
whether EOS is ignored;
-
concurrency;
-
request arrival pattern;
-
warmup procedure;
-
whether prefix caching is enabled and useful;
-
whether speculative decoding is enabled;
-
whether quantization is used;
-
throughput and latency percentiles.
Then repeat runs.
GPU benchmark numbers can move because of thermal state, background processes, compilation/capture warmup, cache state, request ordering, and software versions.
A single screenshot of “tokens/s” is not an experiment.
Three experiments that reveal more than one headline number
After the basic comparison, perform three sweeps.
1. Concurrency sweep
Hold prompt and output lengths fixed.
Measure concurrency 1, 4, 16, 32, 64, and higher if the GPU remains healthy.
This tells you where batching begins helping and where throughput saturates.
2. Context-length sweep
Hold concurrency fixed.
Increase prompt length from short prompts to realistic production contexts.
This exposes the cache-capacity side of serving.
3. Output-length sweep
Hold input length fixed.
Try 32, 128, 512, and perhaps 1,024 generated tokens.
Longer generation places more emphasis on decode and continuously increases each active sequence’s cached state.
Together, these sweeps tell you far more than a single “X times faster” number.
They map the serving engine’s operating envelope.
The production lesson
The deepest lesson from vLLM is not that one library is fast.
It is that LLM inference is a systems problem.
A model’s theoretical compute cost does not determine how much useful work a production GPU will deliver.
You also have to manage:
-
dynamic memory;
-
request lifetimes;
-
cache growth;
-
fragmentation;
-
batching opportunities;
-
prefill/decode interference;
-
tail latency;
-
hardware utilization.
PagedAttention attacked one of the most important constraints by giving KV memory an indirection layer reminiscent of virtual memory.
That, in turn, gave the scheduler room to keep many more sequences alive and continuously build efficient batches.
So when somebody says serving throughput “jumped 10×,” the useful question is not:
Which attention kernel is 10× faster?
It is:
What bottleneck prevented the GPU from processing enough useful work before?
For vLLM’s breakthrough, the answer was heavily tied to KV-cache memory management and the concurrency it unlocked.
Modern vLLM then compounds that advantage with a much larger collection of scheduling and execution techniques.
Your next step
Run the benchmark instead of repeating the headline.
Start with concurrency one. Then sweep concurrency, prompt length, and output length. Record the exact software and hardware configuration, watch for KV-cache preemption, and plot output tokens per second against latency.
Then replace the tiny demonstration model with the model you actually intend to serve and replace synthetic lengths with the distribution from your real application.
That is where PagedAttention stops being an interesting systems paper and becomes a capacity-planning tool: measure how many useful tokens your own GPU can deliver under your own latency budget, and optimize the serving system around that evidence.