Context windows, KV-cache, and why long prompts get expensive

LEARN · LLMS & TRANSFORMERS

Why long prompts feel “free”—until they don’t

Modern large language models can accept remarkably long context windows. It’s tempting to assume that if a model supports 128K, 200K, or even 1M tokens, you should simply keep appending more information to every request.

In practice, long prompts have two hidden costs:

They increase the amount of computation needed before generation begins.
They consume memory through the model’s KV-cache.

Understanding these two ideas explains why long-context inference becomes slower, more expensive, and harder to scale—even when the model technically supports the context length.

This article explains:

What a context window actually is
How the KV-cache works
Why memory grows with prompt length
How to measure KV-cache memory yourself
Practical strategies to keep inference efficient

What is a context window?

A context window is the maximum number of tokens the model can use while generating the next token.

Those tokens include everything:

System prompt
User messages
Retrieved documents
Tool outputs
Previous assistant responses
Newly generated tokens

For example:

System prompt      500 tokens
Conversation     2,500 tokens
Retrieved docs  15,000 tokens
New answer       2,000 tokens
-----------------------------
Total           20,000 tokens

The model attends to every token that still fits inside the context window.

Larger windows allow:

long conversations
code repositories
books
large RAG systems
legal documents
research papers

But supporting long context is only half the story.

The expensive part is remembering it.

A quick refresher: self-attention

Transformer models process tokens through self-attention.

Conceptually:

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V

Every token produces three vectors:

Query (Q)
Key (K)
Value (V)

During generation, each new token compares itself against all previous keys and values.

Without optimization, this would require recomputing everything at every generation step.

That would be painfully inefficient.

Enter the KV-cache

The KV-cache stores the previously computed Key and Value tensors.

Instead of recomputing attention for the entire prompt every time, the model simply reuses the cached tensors.

Imagine generating token number 10,001.

Without a cache:

Recompute:
Token 1
Token 2
...
Token 10000

With a cache:

Reuse:
K1
V1
K2
V2
...
K10000
V10000

Only compute:
Token 10001

This dramatically speeds up autoregressive generation.

Nearly every production LLM server relies on KV caching.

Why memory grows linearly

The important detail is that every token adds new Key and Value tensors.

That means every token permanently occupies memory until generation finishes.

Very roughly:

KV-cache size ∝

layers
× sequence length
× hidden dimension
× bytes per value

The relationship is approximately linear with sequence length.

Double the context.

Approximately double the KV-cache.

Triple the context.

Approximately triple the memory.

This surprises many engineers because model weights remain constant while inference memory continues growing.

The two phases of inference

Inference has two distinct phases.

1. Prefill

The entire prompt is processed.

Prompt
↓
Tokenize
↓
Forward pass
↓
Build KV-cache

This stage is compute-heavy.

Long prompts make prefill slower.

2. Decode

Now the model generates tokens one at a time.

Each new token:

reads the existing KV-cache
appends one more entry
predicts the next token

Decode is much cheaper than prefill but becomes gradually slower as the cache grows because each token attends over a longer history.

Measuring KV-cache growth yourself

The exact cache size depends on the model architecture, precision, and implementation.

You can still estimate memory growth using a simple Python program.

def estimate_kv_cache_gb(
    layers: int,
    hidden_size: int,
    sequence_length: int,
    bytes_per_value: int = 2,
):
    bytes_total = (
        2
        * layers
        * hidden_size
        * sequence_length
        * bytes_per_value
    )

    return bytes_total / (1024 ** 3)


configs = [
    ("8K", 8_192),
    ("32K", 32_768),
    ("128K", 131_072),
    ("256K", 262_144),
]

layers = 80
hidden = 8192

for label, seq in configs:
    gb = estimate_kv_cache_gb(layers, hidden, seq)
    print(f"{label:>6}: {gb:6.2f} GB")

Example output:

    8K:   2.50 GB
   32K:  10.00 GB
  128K:  40.00 GB
  256K:  80.00 GB

This simplified estimate ignores implementation details like grouped-query attention, tensor parallelism, and quantized KV-cache formats, but it captures the most important takeaway:

KV-cache memory scales approximately linearly with sequence length.

Measuring real GPU memory with PyTorch

If you have a CUDA GPU, you can observe memory allocations directly.

import torch

if not torch.cuda.is_available():
    raise SystemExit("CUDA GPU not available.")

device = "cuda"

torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()

sizes = [
    1_000_000,
    5_000_000,
    10_000_000,
]

buffers = []

for elements in sizes:
    tensor = torch.empty(elements, dtype=torch.float16, device=device)
    buffers.append(tensor)

    current = torch.cuda.memory_allocated() / 1024**2
    peak = torch.cuda.max_memory_allocated() / 1024**2

    print(
        f"{elements:>10,d} fp16 values | "
        f"Allocated: {current:.1f} MB | "
        f"Peak: {peak:.1f} MB"
    )

This does not allocate a real KV-cache, but it demonstrates the same principle:

more stored tensors
more persistent GPU memory
linear growth

Actual inference servers continuously allocate KV tensors as tokens are generated.

Why latency increases

There are two reasons.

Longer prefill

Every prompt token must pass through every transformer layer before generation starts.

A 100K-token prompt naturally requires far more computation than a 5K-token prompt.

Users experience this as higher “time to first token.”

Larger attention history

Each generated token attends over everything that came before.

Even with optimized kernels and KV caching, longer histories increase work during decoding.

The result is gradually lower throughput as contexts become very large.

The hidden cost in production

Long context affects more than latency.

It also limits concurrency.

Imagine a GPU with 80 GB of memory.

If one request requires:

40 GB of model weights
30 GB of KV-cache

Only about 10 GB remains for additional requests and runtime overhead.

A server that comfortably handled dozens of short requests may now struggle to host only a handful of long-context sessions simultaneously.

This is why inference providers pay close attention to prompt length, batching strategies, and cache management.

Practical ways to reduce KV-cache usage

Retrieve only relevant information

Instead of inserting an entire knowledge base into every request:

search first
retrieve only relevant passages
send those passages to the model

Retrieval-Augmented Generation (RAG) exists largely because context is expensive.

Summarize conversations

Rather than carrying every previous message forever:

summarize older exchanges
preserve important facts
discard redundant detail

Many chat systems periodically compress history into concise summaries.

Trim repeated instructions

Repeated boilerplate wastes tokens.

Instead of embedding identical instructions in every retrieval chunk, place shared guidance in the system prompt once.

Stream intermediate work

Agent systems often accumulate tool outputs, logs, and reasoning artifacts.

Only keep information that future steps genuinely need.

Everything else consumes context budget.

Cache prompts where possible

Many inference systems support prompt caching or prefix caching.

If thousands of requests share the same system prompt or document prefix, the expensive prefill phase can often be reused.

This reduces latency and improves throughput for repeated workloads.

Cherry on the cake: the “million-token” surprise

One of the biggest surprises in long-context models is that supporting an enormous context window does not mean every request should use it.

Recent production deployments and benchmark reports from long-context model providers have consistently shown that extremely long prompts can increase time to first token by an order of magnitude or more compared with short prompts, even when the generated answer is only a few sentences. In many real-world applications, prefill dominates total latency, meaning users spend most of their time waiting for the model to read rather than write.

The practical lesson is simple: a model advertising a 1M-token context window gives you flexibility, not a recommendation. For many workloads, carefully retrieving or summarizing information yields a faster, cheaper system than filling the available context.

Common misconceptions

“The model weights get larger with longer prompts.”

False.

Model weights stay exactly the same.

Only the KV-cache grows.

“KV-cache eliminates attention cost.”

False.

The cache avoids recomputing previous Keys and Values.

New tokens still attend over the cached history.

“Long-context models remove the need for RAG.”

Not usually.

Even with enormous context windows, retrieval often remains:

cheaper
faster
easier to scale
more memory efficient

“More context always improves quality.”

Not necessarily.

Large prompts may include irrelevant or conflicting information that distracts the model. Carefully selected context often outperforms simply adding more tokens.

Key takeaways

A context window is the total number of tokens available to the model.
Every processed token adds Key and Value tensors to the KV-cache.
KV-cache memory grows approximately linearly with sequence length.
Longer prompts increase both memory consumption and latency.
Prefill cost often dominates long-context inference.
Efficient retrieval, summarization, and prompt caching are essential techniques for production systems.
Supporting massive context windows is a capability—not a signal that every prompt should be massive.

Call to action

The next time you profile an LLM application, don’t stop at counting model parameters. Measure prompt length, monitor KV-cache memory, and track time to first token separately from generation speed. Those metrics often reveal the biggest optimization opportunities—and can dramatically reduce both latency and infrastructure costs.