Understanding FlashAttention Pt 1: Personal Notes
This technical guide explains the mechanics of FlashAttention, an algorithm that optimizes transformer models by reducing GPU memory traffic. It details how tiling, online softmax, and recomputation allow for faster processing without sacrificing mathematical accuracy.
Why it matters
FlashAttention is a foundational optimization for modern AI, enabling the training and inference of larger, more efficient language models.
How IO-Aware Attention Makes Transformers Faster Without Approximating Attention
The mechanism, in three words: Tiling + Online Softmax + Recomputation . Everything in this handbook is elaboration on that summary.
A technical handbook on exact tiled attention: GPU memory traffic, online softmax, forward and backward passes, IO complexity, the evolution from FlashAttention-1 through FlashAttention-4, and current framework behavior.
This handbook was inspired by this tweet . Before the fix, here is what the standard attention implementation looks like. Load $Q, K, V \in \mathbb{R}^{N \times d}$ in HBM, then:
What stands out to me is the number of round trips to HBM. Every intermediate value — $S$, $P$, $O$ — has to be written out and read back. That is the problem FlashAttention is solving.
The handbook itself frames the subject as easiest to understand when three different questions are kept separate:
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in