WebLLM: high-performance in-browser LLM inference engine
WebLLM is a new high-performance inference engine that allows large language models to run directly within web browsers using WebGPU acceleration. It offers full compatibility with the OpenAI API, enabling developers to build private, serverless AI applications.
Why it matters
It democratizes access to powerful AI models by removing the need for expensive server-side infrastructure and enhancing user privacy.
High-Performance In-Browser LLM Inference Engine.
Documentation | Blogpost | Paper | Examples
WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.
WebLLM is fully compatible with OpenAI API . That is, you can use the same OpenAI API on any open source models locally, with functionalities including streaming, JSON-mode, function-calling (WIP), etc.
We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration.
You can use WebLLM as a base npm package and build your own web application on top of it by following the examples below. This project is a companion project of MLC LLM , which enables universal deployment of LLM across hardware environments.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in