Why Is Speculative Decoding Fast?
Speculative decoding improves LLM inference speed not by reducing total compute, but by shifting the GPU workload from memory-bound to compute-bound operations. By using a smaller draft model to propose tokens that a larger model verifies in parallel, the system optimizes hardware utilization.
Why it matters
Understanding the hardware-level mechanics of speculative decoding is essential for developers looking to optimize large language model performance and latency.
It's not because you do less work. It's because you change the kind of work the GPU is doing.
There is a common misconception that speculative decoding is fast because you do less work per token. That just "verifying" a draft requires fewer FLOPs than generating the same tokens.
One of the counterintuitive things about speculative decoding is that you actually do more work! The total number of floating point operations (FLOPs) you execute goes UP for the same sequence.
But then why is it fast? What exactly does an accurate draft model buy you, if it's not compute efficiency?
The answer lies in the type of work your GPU is doing.
Speculative Decoding is a technique where you use a small draft model to propose a likely draft sequence, and then you use your big model to verify that sequence in parallel (in a single forward pass).
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in