WebLLM runs LLMs directly in the browser with WebGPU and an OpenAI-compatible API

The WebLLM, a project from the MLC-AI team with 18.7 thousand stars on GitHub, is an inference engine that runs language models entirely inside the browser, with hardware acceleration via WebGPU and no server in the middle. The proposal is straightforward: you install an npm package, choose a model, and the user's browser starts doing inference locally. No API key, no per-token cost, no data leaving the machine.

For those building AI software in Brazil, where the API bill in dollars weighs heavily and latency to US-based servers is real, this is a piece that changes the architecture calculus for certain products.

What exactly it does

WebLLM brings inference to the client using WebGPU to accelerate the model's operations. All processing happens in the browser, which means privacy by design (prompts don't travel) and offline operation once the model has been downloaded.

The list of natively supported families includes:

| Family | Models mentioned | |---|---| | Llama | Llama 3, Llama 2, Hermes-2-Pro-Llama-3 | | Phi | Phi 3, Phi 2, Phi 1.5 | | Gemma | Gemma-2B | | Mistral | Mistral-7B-v0.3, OpenHermes-2.5-Mistral-7B, among others | | Qwen | Qwen2 0.5B, 1.5B, 7B |

The project is a companion to MLC LLM, and accepts custom models in MLC format, just point to the URLs of the artifacts (model) and the WebAssembly library (model_lib).

The detail that matters most: compatibility with the OpenAI API

The point that most reduces friction for developers is that WebLLM is fully compatible with the OpenAI API. Streaming, JSON mode, seeding for reproducibility, and function calling (still under development) are available through the same interface you already know. In practice, the calling code changes little:

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC";
const engine = await CreateMLCEngine(selectedModel, {
  initProgressCallback: (p) => console.log(p),
});

const reply = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful AI assistant." },
    { role: "user", content: "Hello!" },
  ],
});
console.log(reply.choices[0].message);

An important gotcha: the model parameter of create() is ignored. Model switching happens in CreateMLCEngine(model) or in engine.reload(model), not in the completion call. For streaming, just pass stream: true and iterate over the returned AsyncGenerator.

Installation follows the usual flow, with npm/yarn/pnpm or directly via CDN with import * as webllm from "https://esm.run/@mlc-ai/web-llm", which lets you prototype on CodePen or JSFiddle without a build step.

Not freezing the UI and not reloading the model on every visit

Running an LLM on the main thread would freeze the interface. That's why WebLLM offers support for Web Worker (CreateWebWorkerMLCEngine) to push the heavy computation to another thread, and for Service Worker (CreateServiceWorkerMLCEngine) to avoid reloading the model on every visit and improve the offline experience. The documentation warns that the service worker's lifecycle is managed by the browser and can be terminated at any time, so the application needs its own error handling, with a heartbeat to keep the thread alive.

There is also support for Chrome extensions, with examples of a persistent extension using a service worker.

Cache, integrity, and the first load

The practical point of attention is the first download: loading the model takes a considerable amount of time on the first run without cache, which is why initProgressCallback exists, so you can show progress to the user. After that, WebLLM stores the weights locally through four configurable cache backends in AppConfig.cacheBackend: the browser's Cache API (default), IndexedDB, OPFS (Origin Private File System), and an experimental cross-origin option that depends on a Chrome extension.

For those concerned about the security of the artifact chain, the project added optional integrity verification via SRI hashes (Subresource Integrity). With the integrity field in a ModelRecord, WebLLM checks config, WASM, and tokenizer against the hashes before loading, throwing IntegrityError in case of a mismatch (or just logging it, if onFailure: "warn"). The hashes are generated with openssl:

openssl dgst -sha256 -binary <file> | openssl base64 -A | sed 's/^/sha256-/'

The real bottleneck: user hardware

The discussion in the Hacker News thread revolved around the obvious question: who has the hardware for this? The comment from brucethemoose2 provoked by pointing at the price, upon reading that "the latest MacBook Pro can have more than 60G+ unified GPU RAM," he replied: "...for $3.5K minimum, according to the Apple website :/".

One of the maintainers, junrushao1994, came in to clarify the actual requirement:

To clarify, running this WebLLM demo doesn't need a 3.5k MacBook Pro which costs $3.5k :-) WebGPU supports multiple backends, besides Metal on Apple Silicon, it offloads to Vulkan, DirectX, etc. (...) Our model is int4 quantized, and it is 4G in size, so it doesn't need 64GB memory either. Somewhere around 6G should suffice.

>

-- junrushao1994

Other reports confirmed that it runs on modest machines: hongkonger reported that "both Web LLM and Web Stable Diffusion demos work on my Intel i3-1115G4 laptop with only 5.9GB of shared GPU memory". Still, the int4-quantized model at around 4 GB means a heavy download for the reality of Brazilian internet connections, something worth weighing before assuming every user will have a good experience.

Another technical observation came from bhouston, who noted that the WebNN API (a W3C standard aimed at neural networks, more energy-efficient than WebGPU) is on its way, but is taking so long that the ecosystem ends up "misusing graphics APIs to do NN again". In other words: WebGPU is the bridge available today, not necessarily the final architecture.

What changes for those building

WebLLM opens up a concrete path for AI features that don't need to leave the device, autocomplete, classification, summarization, support chat, structured JSON extraction, without sending a single token to a server or paying per call. For products with strong privacy requirements (healthcare, legal) or to reduce the operational cost of secondary features, it's a real alternative. The trade-off is clear: you exchange the API bill for the requirement of WebGPU and a hefty initial download. It doesn't replace GPT-4 for heavy tasks, but for the core of many apps, it runs in the browser your user already has open.