← Blog
#AI#LLM#WebGPU#Frontend#Edge AI

1,400 Tokens/sec in the Browser: What Liquid AI's WebGPU Demo Means for Client-Side AI

webhani·

A small model, a fast browser

A demo from the webml-community made the rounds recently: Liquid AI's LFM2.5 230M — a 230-million-parameter language model — running entirely inside a browser tab at roughly 1,400 tokens per second on an M4 Max, powered by hand-written WebGPU kernels. No backend call, no API key, no round trip.

In-browser LLM inference isn't new — projects like WebLLM have been running quantized models client-side via WebGPU for a while. What's notable here is the combination: a model architecture (Liquid Foundation Models) specifically built for efficiency at small parameter counts, paired with custom kernels that squeeze meaningfully more throughput out of WebGPU than the generic backends most inference libraries ship with. The result is speed that actually supports real-time UX, not just a proof of concept.

Why this is worth your attention as a web team

Running inference in the browser changes the cost and latency equation for a specific class of features:

  • Zero marginal API cost. Once the model is cached client-side, there's no per-request bill and no rate limit to plan around.
  • No round-trip latency. For short, interactive tasks — autocomplete, quick classification, in-page suggestions — skipping the network hop matters more than raw model quality.
  • Data stays on the device. For anything touching sensitive input (drafts, form content, internal notes), not sending it to a server is a real privacy and compliance simplification, not just a nice-to-have.
  • Works offline once cached. A PWA or offline-tolerant app can keep a narrow AI feature functional without connectivity.

Where it doesn't fit

A 230M-parameter model is not a general-purpose assistant, and it's important not to oversell it internally as one. It's well suited to narrow, well-scoped tasks: text classification, short-form autocomplete, keyword extraction, simple rewriting, lightweight intent detection. It is not a substitute for a frontier model doing open-ended reasoning, multi-turn conversation with long context, or anything where output quality has to hold up without a human reviewing it.

There are also real constraints to plan around: WebGPU support isn't universal (older browsers and some mobile devices need a fallback path), device memory varies widely, and first-load model download size is a real UX cost you have to design around — typically solved with caching and a loading state, not by pretending it's instant.

A minimal example

Here's the shape of a feature-detected setup using Hugging Face's Transformers.js, which supports a WebGPU execution provider out of the box, falling back to WebAssembly when it isn't available:

import { pipeline } from "@huggingface/transformers";
 
async function createLocalAssistant() {
  const hasWebGPU = typeof navigator !== "undefined" && "gpu" in navigator;
 
  const generate = await pipeline("text-generation", "onnx-community/example-small-lm", {
    device: hasWebGPU ? "webgpu" : "wasm",
    dtype: "q4", // quantized weights keep download size and memory reasonable
  });
 
  return async (prompt) => {
    const result = await generate(prompt, { max_new_tokens: 64 });
    return result[0].generated_text;
  };
}

Swap in whichever small, WebGPU-compatible model fits your task — the point is the pattern: detect capability, choose a backend accordingly, and quantize aggressively so the download and memory footprint stay reasonable for a browser tab.

webhani's take

We'd treat this as a good fit for prototyping specific, narrow client-side AI features where cost sensitivity or data privacy is the driving constraint — in-app search re-ranking, lightweight content moderation before a message is even sent, or form-fill suggestions. It's not a reason to move your core product AI features off a server-side model; quality and consistency still favor larger models running where you control the environment.

Build a server-side fallback from day one rather than treating WebGPU support as guaranteed, and be precise with stakeholders about what a 230M model can and can't do — the demo's speed is genuinely impressive, but speed isn't capability. Used for the right narrow task, it's a real cost and latency win; used as a drop-in for your product's main AI feature, it will disappoint.

References: WebLLM: A High-Performance In-Browser LLM Inference Engine (mlc-ai), 1,400 Tokens Per Second in Your Browser (Essa Mamdani)

← Back to Blog