You could have invented transformers from memory
This article explores the theoretical foundations of how large language models store and recall information, comparing linear maps, MLPs, and self-attention mechanisms. It provides a framework for understanding memory capacity and interference in neural networks.
Why it matters
Provides a deeper technical understanding of the mechanisms that allow AI models to function as effective knowledge bases.
Large language models today can recall obscure facts from every corner of the world. Most recently, I was running late for a friend's board game night, so I asked Astra to remind me the rules to Dune: Imperium. It gladly narrated them to me as I drove over and helped me remember the rules well enough for me to play again. We're so used to asking models about anything that we take their vast knowledge for granted.
How do models remember so much information? And how much can they store?
In this post, I'll go through an intuitive way to look at linear maps, MLPs, and self-attention as different kinds of memory stores. From this intuition we can derive some heuristics for how much memory each layer can store.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in