Run Gemma on-device with LiteRT-LM¶

LiteRT-LM is Google AI Edge's runtime for running language models directly on a device — Android, iOS, macOS, Windows, Linux and Raspberry Pi — with CPU, GPU and NPU acceleration and no network round trip.

Install¶

uv tool install litert-lm

Run¶

Weights are pulled straight from the Hugging Face LiteRT Community:

litert-lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --prompt="What is the capital of France?"

Multi-Token Prediction¶

LiteRT-LM supports MTP, which speeds up decoding with no quality loss: up to 2.2x on mobile GPUs and 1.5x on mobile CPUs.

Google recommends MTP for every task on GPU backends, and for E4B on CPU. On E2B on CPU it helps most with rewriting, summarizing and coding, but can slow down freeform generation slightly — so enable it selectively there.

Deploy your own fine-tune¶

Export Safetensors — including a model you have fine-tuned — to the .litertlm format:

uv tool install litert-torch-nightly

litert-torch export_hf \
  --model=google/gemma-4-E2B-it \
  --output_dir=/tmp/gemma4 \
  --externalize_embedder \
  --jinja_chat_template_override=litert-community/gemma-4-E2B-it-litert-lm

Then run the exported bundle the same way:

litert-lm run /tmp/gemma4/model.litertlm --prompt="What is the capital of France?"

Model sizes¶

Google publishes measured prefill, decode and time-to-first-token figures per device and backend on the Gemma 4 LiteRT-LM page.

Platform guides¶

LiteRT-LM has APIs for the CLI, Python, Android (Kotlin), iOS (Swift) and C++. See the LiteRT-LM documentation for each.

To try Gemma on a phone without writing code, use the AI Edge Gallery.