Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes
A developer successfully trained a 3.8 billion parameter language model from scratch for under $1,000 using cloud-based GPU resources. The project highlights how disciplined software engineering and efficient infrastructure allow individuals to achieve research-grade results outside of major corporate labs.
Why it matters
This demonstrates the democratization of AI development, showing that high-performance model training is becoming accessible to independent researchers.
Somewhere between “nanoGPT toy” and “you need a research lab” there’s a large, under-described region where one person with a few thousand dollars can train a meaningful model.
I wanted to see language and understanding emerge from random weights for myself, and to learn the parts you can only learn by starting from scratch. This project was written in the evenings, debugged on a 5090 and finished on rented B200s. It was heavily inspired by Andrej Karpathy’s nanochat .
The result is a 3.8B-parameter model scoring 0.384 on CORE , trained on 65B tokens in 43 hours for $998 .
What follows is what worked, what didn’t, and what I still don’t know.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in