Prefill vs Decode: Why Your Local Model Feels Fast or Slow
The performance of large language models is split into two distinct phases: prefill and decode, which have different hardware requirements. Prefill is compute-bound and handles the initial prompt, while decode is memory-bound and generates tokens one by one, explaining why some models feel slow during generation.
Why it matters
Understanding these technical bottlenecks is crucial for developers and users to optimize hardware configurations for AI applications.
Ask two people running the same model on the same box whether it is fast and you can get opposite answers.
The first pastes 40,000 tokens of a codebase and waits. Nothing happens for several seconds. The machine feels broken.
The second types a two-line question and watches the answer stream out immediately. The machine feels excellent.
Both are describing the same hardware accurately. They are just reading different clocks, because a language model answers you in two phases that have almost nothing in common.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in