How continuous batching speeds inference

The waste in static batching

Static batching assembles a fixed group of requests, runs them together, and waits for every sequence in the batch to finish before admitting anything new. A batch of 32 requests averaging 50 tokens per second is held back by the one generating at 30. That slower sequence occupies a GPU slot for its entire lifetime, and after it finishes, its memory sits idle because the batch boundary has not arrived. On bursty production traffic with variable-length outputs, static batching routinely leaves 30-50% of GPU throughput on the table.

How continuous batching works, and the three queues

Continuous batching — iteration-level scheduling, in-flight batching — decouples admission from batch boundaries. The scheduler re-evaluates on every forward pass: which sequences run, which new ones join, which finished ones leave. When a request completes, its slot goes to the next waiting sequence immediately. The output tokens are identical to static batching; only the GPU utilization changes. Under the hood, most implementations organize sequences into three buckets:

• Waiting. New requests queued in arrival order, with optional fairness or priority policies layered on top.
• Running. Sequences generating tokens this forward pass. They hold KV-cache slots in GPU memory.
• Swapped. Preempted sequences whose KV cache was evicted to CPU RAM. They wait until memory frees up, then resume.

Preemption: when the batch outgrows memory

When the combined KV cache of running sequences exceeds GPU memory, the scheduler must evict something. Swap moves a sequence's KV cache to CPU RAM and restores it later — fast resume, costs PCIe bandwidth. Recompute discards the KV cache entirely and regenerates it from prompt tokens on reschedule — saves bandwidth, costs compute. Swap is usually preferred for latency-sensitive workloads; recompute makes sense when PCIe is the bottleneck. Either approach keeps the system stable under load instead of crashing with an OOM.

Where the 3-10× comes from

The throughput multiplier over static batching depends on workload shape. Uniform, same-length requests see a modest 2-3× gain because static batching already does reasonably well. Bursty, mixed-length production traffic with unpredictable completion times lands at the high end — 5-10× — because static batching spends more and more GPU time waiting on the longest sequence. Most real-world serving workloads lean toward the high end, which is why continuous batching is now table stakes for any serious inference stack.

Why single-tenant means a better scheduler

On shared GPU infrastructure, your continuous-batching scheduler competes for HBM bandwidth and SMs with every other tenant on the node. The scheduler sees only your requests, but the hardware does not cooperate. Dedicated GPUs eliminate that contention — the scheduler owns the full memory bandwidth, the full KV-cache pool, and all the tensor cores. Techniques like speculative decoding and paged attention amplify these gains on isolated hardware. See our benchmarks for measured throughput from dedicated nodes.