Edge AI on Raspberry Pi 5 with LiteRT and Gemma: what Google's guide shows

O Google publicou um guia rodando a família Gemma via LiteRT num Pi 5, totalmente offline. Fui atrás da arquitetura e dos números pra ver o que serve pro maker e IoT no Brasil.

Google published a guide running the Gemma family via LiteRT on a Pi 5, fully offline. I dug into the architecture and the numbers to see what's useful for makers and IoT in Brazil.

Edge AI is no longer a slide promise. In August 2026, Google published a guide showing the Gemma family running via LiteRT on a Raspberry Pi 5, fully offline, and I dug in to understand what the architecture promises for people building things with cheap hardware here in Brazil.

A heads-up before you go copying commands: the model names and CLI commands that appear in this text are exactly the ones Google's post brings. I didn't reproduce the end-to-end pipeline on a physical Pi, so I treat the numbers for what they are (metrics reported by Google) and recommend checking the official documentation and the repository in the LiteRT community on Hugging Face before downloading anything. Package and repository names change fast in this ecosystem, and a 404 in the middle of setup is guaranteed frustration.

Editor's note: we could not confirm that the pip package "litert-cli" and the exact syntax of the "litert lm run" command cited below exist as written; Google AI Edge's official LiteRT tools circulate under other names (such as "ai-edge-litert" on PyPI or via a custom build of the LiteRT-LM repository). We also could not confirm the "Gemma 4" spelling for the E2B/E4B suffixes cited in the text; the public naming for these models is "Gemma 3n". Treat package, repository, and command names as possibly outdated, and always check the official documentation before running anything.

Why the edge matters (for real)

Running a model locally on the Pi eliminates three classic pain points: network latency, cloud dependency, and data leakage. For IoT, social robotics, and "smart" cameras, this isn't a luxury, it's a requirement. No stable internet in the warehouse? No problem. Sensitive data that can't leave the device? It stays on the device.

The key point is LiteRT, Google AI Edge's on-device inference runtime, which runs both classic ML models and LLMs. On top of it sits LiteRT-LM, an orchestration layer focused on language models, with CPU acceleration via XNNPACK.

The Gemma family sized for the Pi

Google lists variations designed for hardware constraints. The smaller ones serve as a base for specific tasks:

• Gemma 3 270M: compact base for task-specific fine-tuning (sentiment analysis, entity extraction).
• EmbeddingGemma 300M: on-device embeddings, great for RAG and semantic search.
• Gemma 3 1B: lightweight multilingual text, good for summarization and content generation.

The post also cites two models with the E2B and E4B suffixes, which Google spells as "Gemma 4". The idea behind these suffixes is interesting: they're models with memory-mapped per-layer embeddings to save RAM, something critical on a constrained device. E2B is presented as the focus for continuous monitoring and fast text/image/audio inference; E4B, as the "sweet spot" between reasoning and size. Before downloading, check the exact repository name in the LiteRT community on Hugging Face, because that's where the source of truth for the model's current spelling lives.

The numbers that matter

This is where I separate hype from reality. According to Google, on a Pi 5 the E2B model delivers:

• 99 tokens/s on prefill
• 9 tokens/s on decode
• peak memory of 1432 MB

The text also cites an efficient tokenizer (~4.2 characters per token) and an end-to-end generation of ~27.3 characters/s in the Reachy Mini voice demo, close to 300 words per minute (Google compares it to normal human speech, ~150 wpm). A critical look is worth it: 9 tokens/s at 4.2 characters per token would give ~37.8 characters/s, not 27.3. Either there's unexplained pipeline overhead, or the numbers were measured in different scenarios. Either way, the order of magnitude supports the argument: real-time translation and voice response on a Pi is doable. It's not a frontier model running locally, but for focused tasks it gets the job done.

CPU or GPU: the move is to split

An architecture detail worth highlighting. The Pi 5's quad-core Cortex-A76 CPU delivers ~153.6 GFLOPS (FP32) and up to ~2.0 TOPS (INT8); the integrated VideoCore VII GPU, clocked at 800 MHz, sits at ~76.8 GFLOPS (FP32) and ~0.24 TOPS (INT8). The GPU is much weaker, but Google enabled inference on it via a WebGPU backend (which runs on top of Vulkan on Linux) through ML Drift.

The trick isn't choosing one or the other, it's parallelizing. In the Reachy Mini robot's pipeline, object detection (Ultralytics YOLO) runs continuously on the GPU, while speech recognition (Moonshine), reasoning (the Gemma LLM), and TTS stay on the CPU. This way vision doesn't fight for cycles with the LLM, what Google calls heterogeneous parallel execution, and the system keeps thermal efficiency, which matters a lot on a Pi.

Setup: the path Google indicates

The post presents a LiteRT CLI that bundles conversion, quantization, benchmarking, and inference into a single set of commands, instead of you piecing together several libraries by hand. The suggested installation is via pip, preferably in a virtualenv:

pip install litert-cli

Then, the example runs the model straight from Hugging Face, passing your authentication token:

export HUGGING_FACE_HUB_TOKEN=<seu_token_aqui>
litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --attachment=image.jpg \
  --prompt="You are Reachy Mini. Identify the main object in front of you, state its location (Left/Right/Center), and suggest head action in 10 words or less."

My recommendation: before reproducing this, check the official LiteRT documentation (Google AI Edge developer site) to see if the package name and syntax match the post, and check the exact model repository name in the LiteRT community on Hugging Face. Package and repo names are exactly what varies the most between an announcement and the published version.

The spirit of the example, that's what's worth keeping: the model receives an image, identifies the main object, states the position (left/right/center), and suggests an action, all in a few words. It's the Reachy Mini's brain condensed into a single multimodal call.

Lean footprint and what's coming

For IoT, binary size is money. Google argues that generic AI runtimes drag along heavy desktop/server dependencies, while LiteRT is modular and designed for on-device deployment, which makes a difference in disk and memory on constrained devices.

The post also announces future integration with Hailo accelerators, promising inference offload to the Raspberry Pi AI HAT+ and AI HAT+ 2 using the same LiteRT workflow. It's a roadmap, not something to use today, but it indicates that whoever builds the pipeline now tends to gain hardware acceleration later without rewriting everything.

Think of the guide as an architecture map, a great one, more than as a tutorial you can copy line by line. The concept (parallelizing CPU/GPU, running a lightweight LLM offline on a Pi) is solid and reproducible; it's the command surface that calls for double-checking against the official source before the git clone.