Recurrent Looped Transformer
This article introduces the Recurrent Looped Transformer (RLT), a new AI model architecture that combines a causal encoder with a recurrent decoder. RLT aims to achieve unbounded temporal depth for latent reasoning while optimizing hardware efficiency and co-design with RL algorithms.
Why it matters
RLT represents a significant advancement in AI model design, potentially enabling more complex and continuous reasoning in large language models. This could lead to improved efficiency and scalability for various AI applications, including reinforcement learning.
Recurrent Looped Transformer (RLT) combines a causal encoder with a recurrent decoder that carries its final hidden state and layerwise sliding-window attention (SWA) cache across every prompt and response token. The encoder constructs global key–value memory; the decoder extends a continuous latent computation as the sequence grows.
The design brings together latent reasoning with unbounded temporal depth , model–hardware co-design , and model–RL algorithm co-design . Parallel encoder work, sequence batching, memory reuse, and checkpointing surround a recurrent core. Pretraining, SFT, sampling, and current-policy replay share the same complete-state transition.
Infinite depth refers to an extensible temporal path, not infinite work within a token. Realized reasoning gains, hardware efficiency, and RL scaling remain to be established.
Each token extends the recurrent path through the full decoder. After \(t\) tokens, that path traverses \(tL_D\) decoder blocks while the per-token block count stays fixed.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in