Speculative Decoding & Speedups

Decode is serial and memory-bound: each token costs a full weight read, and the GPU idles through it. Speculative decoding exploits the idle arithmetic.

A tiny draft model — say 100M parameters — proposes the next 4–5 tokens (cheap: its weight reads are small). Then the big model scores all 5 in parallel in a single forward pass: attention is already shaped to evaluate every position at once. Accept the longest prefix that matches what the big model would have sampled; restart from the first rejection.

The magic: that verification pass reads the big weights once — the same cost as generating one token. Typical result: 2–3× faster decoding, and it's exact — the output distribution is provably identical to normal sampling.