How to build a diffusion language model
This article explores the shift from traditional autoregressive language models to diffusion-based models for text generation. It explains the technical mechanics of how diffusion models refine sequences iteratively rather than token-by-token.
Why it matters
Diffusion models represent a potential paradigm shift in generative AI, offering different trade-offs in speed, quality, and context handling compared to current industry standards.
How to Build a Diffusion Language Model An introduction to diffusion language models and the research advances that underlie today's diffusion LLMs. We describe the building blocks of recent open-source models, starting from simple masking diffusion, and including techniques for iterative refinement, post-training, and variable-length generation. Material is adapted from workshop talks and lectures at ICLR 2026 and MLSS 2026 .
Two families of generative AI algorithms are widely used today. For continuous data such as images or video, the state-of-the-art approach is based on diffusion models . For discrete data such as text or code, the standard approach is instead autoregressive models . This article explores an alternative for discrete data, one built on the modern paradigm of diffusion.
Mainstream language models are autoregressive: they generate tokens left-to-right, one at a time, each conditioned on the tokens before it. This approach is powerful, but it also has inherent limitations:
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in