CS336: Language Modeling from Scratch

Tatsunori Hashimoto Instructor Percy Liang Instructor Herman Brunborg CA Marcel Rød CA Steven Cao CA Logistics Lectures: Monday/Wednesday 3:00-4:20pm in Skilling Auditorium Recordings: YouTube playlist Office hours: Percy Liang: Fridays 11am-12pm in Gates 366 Tatsu Hashimoto: Tuesdays 11-12am in Gates 364 Marcel Rød: Tuesdays 4:30-5:30pm in Gates 498, Wednesdays 4:30-5:30pm in Gates 415 Herman Brunborg: Wednesdays 1:30-2:30pm, Fridays 1:30-2:30pm, location Gates 392 Steven Cao: Mondays 4:30-5:30pm, Thursdays 9:30-10:30am, Gates 200 Contact : Students should ask all course-related questions in public Slack channels. All announcements will also be made in Slack. For personal matters, email cs336-spr2526-staff@lists.stanford.edu . Content What is this course about? Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment.