KV cache & paged attention deep dive

What the KV cache is, and why it dominates memory

During autoregressive generation, each new token attends to every previous token in the sequence. Without a cache, the model would recompute keys and values for the entire prefix on every forward pass — O(n²) work that becomes catastrophically expensive as context grows. The KV cache stores the key and value tensors for every processed token, so each new token only computes attention against the cached history. The cost is memory. For a model with L layers, Hkv key-value heads, and head dimension D at b bytes per element, the KV cache for one token is `2 × Hkv × D × b × L` bytes. That scales linearly with context length and quickly dwarfs the model weights.

Why contiguous allocation fails

The naive approach allocates a single contiguous block of GPU memory per sequence, sized for its maximum context length. Since most sequences never reach the maximum, 60-90% of that allocated memory sits empty. Worse, the allocator must find a single contiguous free region — fragmentation builds up and you run out of usable memory long before you exhaust total free bytes. This is the bottleneck that historically capped concurrency on early inference servers. A 70B model on an 8×H100 node might serve 20 concurrent sequences under contiguous allocation; the same hardware with PagedAttention can serve 60-80.

PagedAttention: virtual memory for attention state

PagedAttention, introduced by vLLM and now adopted across the serving ecosystem, treats the KV cache like virtual memory. Instead of one large contiguous buffer per sequence, it allocates fixed-size blocks — typically 16 tokens each. A block table maps each sequence's logical token positions to physical blocks anywhere in GPU memory. When a sequence needs more KV, the scheduler allocates a new block from the free pool — no fragmentation, no pre-reserving memory that might never be used. Waste drops from 60-90% to roughly 4%, the unused tail of the last block. Throughput jumps 2-4× because far more concurrent sequences fit in the same HBM.

What 16-token blocks buy you

The block size is a deliberate engineering choice. Too small — say 4 tokens — and the block-table overhead dominates, plus attention computation loses efficiency because tensor cores prefer contiguous work. Too large — say 128 tokens — and the tail waste creeps back up. At 16 tokens, you balance three things:

• Low waste. Across thousands of sequences of varying lengths, the average unused tail is ~8 tokens — roughly 4% of total KV memory.
• Efficient attention. 16-token blocks give attention kernels enough contiguous data to keep tensor cores fed.
• Simple preemption. Evicting a sequence means freeing its blocks to the pool. Swapping to CPU RAM is block-by-block, not a contiguous reorganization.

Prefix caching: share the system prompt

When multiple requests share a common prefix — a long system prompt, a few-shot example block, a document preamble — PagedAttention can share the KV blocks for that prefix across all sequences. Only one copy of the prefix KV cache sits in memory; every request maps those blocks into its own block table. For chatbot workloads with verbose system prompts, this can cut KV memory consumption by 40-60% at high concurrency.

KV quantization cuts memory in half

KV-cache tensors can be stored in FP8 instead of FP16, halving their memory footprint. The quality impact on long-context tasks is typically negligible — attention patterns are resilient to reduced precision because the softmax normalizes across the sequence. Combined with PagedAttention, FP8 KV cache roughly doubles concurrency again on the same GPU, or serves twice the context length. For how quantization more broadly affects model output, see our guide to quantization and model quality.

Why this matters for long context

A single 128K-context request on a 70B model can consume more KV-cache memory than the model weights themselves. Without PagedAttention's block allocation, prefix sharing, and quantization, serving long-context workloads at production scale is economically impossible — you would need absurd numbers of GPUs just to hold the KV cache. With these techniques on dedicated hardware, continuous batching and speculative decoding stack on top to deliver 300+ tokens per second even at high context lengths. For how different hardware generations handle these memory pressures, see our benchmarks.