Stop Thinking of LLMs as Next-Token Predictors
This article argues that modern LLMs have evolved beyond simple next-token prediction due to post-training techniques like reinforcement learning with verifiable rewards. It suggests that current models are better understood as goal-oriented agents that learn from outcomes rather than just statistical patterns.
Why it matters
Understanding the true nature of LLM training helps researchers and developers better predict model behavior and improve safety and reasoning capabilities.
Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. It’s a fine zeroth-order approximation, and it is grounded in something real: transformer-based language models emit tokens autoregressively:
while not done : tokens . append ( model . sample_next_token ( tokens )) This certainly has the shape of something you might call a next-token predictor. During pre-training, the model repeatedly takes some prior tokens, looks at the token that actually followed them, and makes that token more likely to be sampled next. Conceptually, the training loop looks something like this:
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in