Speculative Decoding

Speculative decoding is an inference-acceleration technique in which a small draft model proposes several tokens at once and a larger model verifies them in parallel, speeding up generation without changing the output.

Because the large model checks the draft tokens together rather than generating one at a time, throughput rises while the final output matches what the large model would have produced.

It lowers latency and effective inference cost, a lever on the price of running AI labor at scale.