B200 Attention Kernel from Scratch to Near-SOTA in 60 Diagrams
This technical guide provides a step-by-step progression for building a high-performance attention kernel for NVIDIA's B200 GPU. It uses 60 diagrams to explain optimizations that achieve 94.4% of FlashAttention-4 performance.
Why it matters
Optimizing GPU kernels is critical for the efficiency of large-scale AI training and inference on the latest hardware.
Build and understand one of the most complex GPU kernels on the latest hardware
60 diagrams · 14-kernel progression · 94.4% of FlashAttention-4 · CUDA/PTX · video-generation capstone
Percent of same-run Stock FA4 across 14-kernel progression and minor optimizations covered at the end. Each implementation is timed on its preferred contiguous layout; see Benchmark Calibration .
In this blog, we build a dense B200 attention kernel from scratch in CUDA and a little PTX, from baseline to 94.4% of FlashAttention-4 performance on 4K, 8K, and 16K shapes used in the FA4 paper.
The main contribution is the visual guide : a beginner-friendly progression built around 60 diagrams. We first build an intuitive understanding of how the naive kernel works, then add one optimization at a time, with detailed diagrams, concise explanations, and code.
For the capstone project, we plug the final kernel into a video-generation model.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in