PSSA: A non-transformer language model written from scratch in Rust
PSSA is a new, non-transformer language model written in Rust that uses a recurrent state-space layer and episodic memory. It demonstrates faster training and inference speeds compared to standard transformers while maintaining better generalization on test data.
Why it matters
This represents a potential shift in AI architecture, moving away from the computationally expensive transformer model toward more efficient, linear-scaling alternatives.
PSSA is a small language model that is not a transformer. It reads text one token at a time through a recurrent state-space layer, keeps a bank of episodic memories it can look things up in, and rewrites part of its own weights while it runs. It is written in Rust from scratch, with no PyTorch, no TensorFlow, and no ML framework of any kind underneath it.
At matched parameters and on the same corpus, it learns faster than a transformer and generates text about twelve times quicker on the same CPU.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in