Speculative decoding for faster tokens

One token per forward pass is the bottleneck

Autoregressive decoding generates tokens sequentially: each new token requires reading every model weight plus the KV cache from HBM, running a full matrix multiply, and producing a single output. The arithmetic intensity is low — most of the time is spent waiting on memory, not computing. That is why inference is memory-bandwidth bound. Speculative decoding sidesteps this by producing multiple tokens per forward pass without changing the model architecture or the output distribution.

Draft, then verify: the two-step mechanics

Speculative decoding runs two operations in sequence. A small, fast draft model proposes, say, five candidate tokens ahead of the current position. The large target model then verifies all five in a single forward pass — processing them in parallel as a batch. Each candidate is accepted if it matches the target's distribution and rejected if it does not. Accepted tokens are kept and streamed. Rejected ones are discarded, and the target generates one corrected token from that position. The technique is lossless: the output distribution is mathematically identical to running the target model alone, because rejection sampling guarantees it. You get the same text, just faster.

Three ways to generate draft tokens

The draft step can use different approaches, each with its own accuracy-to-overhead tradeoff:

• Small standalone draft model. A tiny model — often the same architecture at a fraction of the parameters — trained to mimic the target. Simple to implement. Accuracy depends on how well the draft predicts the target's next tokens.
• EAGLE draft head. A lightweight prediction head attached to the target's hidden states guesses upcoming tokens. Higher accuracy than a standalone draft because it sees the target's internal representations. Negligible memory overhead.
• Multi-token prediction (MTP). The target model is trained to predict multiple future tokens natively, without an external draft mechanism. DeepSeek-V3 and V4 use this approach. No separate draft model needed, but the base model must support it.

When it helps, and when it does not

Speculative decoding reduces latency most on latency-bound, predictable-text workloads — chat, summarization, code completion, structured output — where the draft model guesses correctly often. It delivers roughly 2-3× more tokens per second on these workloads. On highly creative or unpredictable text, draft accuracy drops and the benefit shrinks, though it never hurts because rejected drafts simply fall back to standard decoding. Critically, the technique does not help throughput-bound batch workloads where you are already GPU-saturated; it is a latency play, not a throughput play.

EAGLE vs standalone draft — which to use

EAGLE-style draft heads consistently outperform standalone draft models in accuracy per unit of overhead because they operate on the target's hidden states rather than guessing from tokens alone. The tradeoff: EAGLE requires modifying the serving stack to attach the draft head, while a standalone draft model works with any unmodified target. For production deployments where every millisecond counts, EAGLE is usually the right call. For quick experiments, a standalone draft model gets you roughly 80% of the benefit with zero integration work.

Speculative decoding on dedicated hardware

The draft model runs on the same GPU as the target, sharing memory bandwidth. On shared infrastructure, that means the draft competes with other tenants for bandwidth — eroding the benefit. On dedicated hardware, both the draft and target have uncontended access to HBM. Combined with continuous batching and paged attention KV management, speculative decoding on an isolated node is part of the stack that delivers 300+ tokens per second on tier-1 open models. For how different hardware generations affect these numbers, see our benchmarks.