How Is Compression Prediction?
This article explores the theoretical equivalence between data compression and predictive modeling, noting that both involve optimizing for probability. It discusses how modern language models are essentially instantiating long-standing information theory principles at a larger scale.
Why it matters
Understanding the mathematical link between compression and prediction is fundamental to the development and evaluation of modern AI models.
Over the past few weeks, I have repeatedly encountered the same claim on Hacker News: compression is prediction . The recent discussion has approached it from both directions. Two 3Blue1Brown videos, Reinventing Entropy and But what is cross-entropy? , derive entropy and cross-entropy from the limits of source coding. An ngrok article follows the same mathematics through arithmetic coding and language models. Salvatore Sanfilippo asks how far the resulting identification between prediction and compression should be taken.
These explanations meet at one fact. A probabilistic model assigns a conditional probability to every possible continuation, and an entropy coder converts the probability assigned to the observed continuation into bits. For a sequence x 1 : n x_{1:n} x 1 : n and a model Q Q Q , the resulting ideal payload length is
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in