Gemma on Android: running an on-device LLM with LiteRT

This post is the hands-on half of my guide to running an LLM locally. Here I zoom in on one concrete case: getting Gemma onto an Android phone so it works offline and your data never leaves the device. If you wonder why that last bit matters so much, I covered it in my post on Shadow AI.

I did it for Balùa, my app that looks at a photo of a plate and tells you which foods are on it. I fine-tuned Gemma to return JSON with the foods, exported it to LiteRT’s format and ran it on a Pixel 8 Pro. It worked, with caveats. Here are the steps, the errors and how long it took.

When I looked for how to do this, most of what I found skipped the failures. So here they are.

What do you need to run Gemma on Android?

You need three things: a Gemma model in .litertlm format, a runtime in the app to load it (LiteRT-LM, or MediaPipe LLM Inference, which is what I used) and an arm64 Android phone with at least 6 GB of RAM. If you won’t fine-tune, download Google’s ready-made conversion and skip the hard part.

The pieces, in order:

• The model. Google publishes converted Gemma models on Hugging Face under litert-community. The Gemma 4 E2B one is about 2.6 GB.
• The runtime. I used MediaPipe tasks-genai (versions 0.10.29 to 0.10.35). Today Google is pushing LiteRT-LM, with a Kotlin API and CPU, GPU and NPU support.
• The device. arm64, 6 GB of RAM or more, and room for the model, which doesn’t go inside the APK.

My advice: if the official model does the job, use it. The official int4 Gemma 3n E2B identified food on my Pixel out of the box. The next section is only needed if you’ve trained your own version.

How do I export Gemma to .litertlm?

You export it with litert-torch, which only ships builds for Linux and macOS. On Windows, use WSL. The keys are pinning transformers to 4.56.2, giving WSL plenty of swap, and having the SentencePiece tokenizer plus a special_tokens_map.json with eoi_token in the model folder. After that it’s a single command.

The litert-torch issues (#994 and #1039) treated exporting Gemma 3n with vision as broken. It can be done. This is the environment that worked for me, exactly as it appears in the cookbook:

uv venv exportenv --python 3.12
UV_LINK_MODE=copy uv pip install 'litert-torch==0.9.1' 'transformers==4.56.2' pillow torchvision timm

The transformers version matters most. With 5.x I had to patch litert-torch because the model’s internal structure changed. The export finished without errors and the model came out broken, generating things like df online df online.... With 4.56.2 those patches aren’t needed.

A uv detail: it uses hard links. If you reuse an environment, reinstall with UV_LINK_MODE=copy so you don’t inherit patches from another one.

Memory. The step that builds the MLIR module for a ~11 GB model ran out of memory with 31 GB of RAM, and WSL killed it with a bare Killed. I fixed it by giving WSL 28 GB of memory and 72 GB of swap in .wslconfig, then restarting with wsl --shutdown.

Two files that must be in the model folder:

1. tokenizer.model (SentencePiece, about 4.7 MB). Gemma 3n’s fast tokenizer only ships tokenizer.json, and then MediaPipe fails with SentencePiece tokenizer is not found in the model. Download it from google/gemma-3n-E2B-it.
2. special_tokens_map.json with eoi_token. If it’s missing, the export dies with KeyError: 'eoi_token'. Copy it from the base model.

And the command:

litert-torch export_hf /ruta/merged_gemma3n /out \
  --task=image_text_to_text --export_vision_encoder \
  --quantization_recipe=dynamic_wi4_afp32

It takes about 30 minutes and produces a model.litertlm of about 2.7 GB with vision. Watch the quantization recipe names, they’re not intuitive:

Recipe	What it does	When to use it
dynamic_wi4_afp32	4-bit weights	First attempt, smallest size
dynamic_wi8_afp32	8-bit weights, roughly 2x the size	If int4 degrades the output
weight_only_wi4_afp32 / wi8	Quantizes weights only	Alternative to the dynamic ones
static_wi8_ai8 / ai16	Weights and activations	When you need to squeeze more

dynamic_int4 doesn’t exist, even though it’s the first thing you’ll type.

How do I wire the model into the app?

Copy the .litertlm to the app’s external files folder, never to /data/local/tmp, and load it with the runtime from a background thread. With MediaPipe the GPU backend crashed my app, so I forced CPU. Keep maxTokens at 1024 or more, because the image alone takes about 258 tokens.

The model doesn’t fit in the APK. During development I push it with adb push to Android/data/<package>/files/, the folder getExternalFilesDir returns. The app reads it without special permissions. From /data/local/tmp the app can’t read it and fails without a clear message.

The MediaPipe configuration I used, from the cookbook:

LlmInference.LlmInferenceOptions.builder()
    .setModelPath(path)                       // en getExternalFilesDir, NO /data/local/tmp
    .setMaxTokens(1024)                        // input imagen ~258 tokens; no bajar
    .setPreferredBackend(LlmInference.Backend.CPU)   // el backend GPU CASCA (SIGSEGV)
    ...

Three things I ran into:

• The GPU backend crashed with a SIGSEGV while creating the OpenCL executor. On the CPU it was stable.
• The model repeated the JSON in a loop. I solved it by parsing with a regex and dropping duplicates instead of expecting clean JSON.
• Android warned that the native library wasn’t 16 KB aligned. On a device with 4 KB pages (check with getconf PAGESIZE) it’s only a warning and the model loads anyway.

If you’re starting today, look at LiteRT-LM before MediaPipe. According to the official docs, you set up an EngineConfig with the model path and the backend (with a separate one for vision), initialise the Engine off the main thread and talk to it through createConversation(). I haven’t tried it in Balùa yet, so I can’t tell you about its failure modes.

A design tip: put the analyser behind an interface. My app had an analyser that fell back to a mock when it couldn’t find the model. That let me build the UI without depending on the model, and later switch models without touching anything else. It’s what I describe in hexagonal architecture and LLMs.

What performance can I expect?

It depends on the task. For text, Google reports around 47 tokens per second on CPU and 52 on GPU for Gemma 4 E2B on a Galaxy S26 Ultra. For a photo with Gemma 3n on the CPU of my Pixel 8 Pro, each analysis took several minutes, with the CPU at 365-421% and around 3 GB of RAM.

These are Google’s numbers for Gemma 4 E2B with LiteRT-LM, taken from its Hugging Face model card (1,024 input tokens, 256 output tokens):

Device	Backend	Prefill (tok/s)	Decode (tok/s)	First token	Memory
Galaxy S26 Ultra	CPU	557	46.9	1.8 s	1,733 MB
Galaxy S26 Ultra	GPU	3,808	52.1	0.3 s	676 MB
My Pixel 8 Pro, Gemma 3n with a photo	CPU (MediaPipe)	n/a	n/a	several minutes per photo	~3 GB

The last row is mine and isn’t a fair comparison: different model, different runtime, an older phone and, above all, an image as input. Encoding the image is expensive. But it’s what I actually saw, and it’s a good reason not to promise instant answers in a multimodal app.

What breaks and how do you catch it?

The conversion is what breaks most: the export finishes fine and the model comes out broken. You catch it by comparing against the bf16 model on the PC and against Google’s official .litertlm, always with two different images. If two different photos get the same answer, the vision encoder is broken.

That was my case. With the export fixed, my fine-tuned Gemma 3n returned flawless JSON. But two different photos got the same list of foods, with the same portions. This is how I isolated it:

Test	Result	What it told me
My model in bf16 on the PC	Perfect	Fine-tuning and prompt were fine
Google’s official .litertlm on the Pixel	It could see: food → foods, portrait → “Man”	The runtime was fine
My exported model, int4	Made-up foods, garbled words	Quantization was degrading it
My exported model, int8	Clean JSON, but the same for any photo	Vision was lost in conversion

There was no plan B inside MediaPipe either: its LoRA support only covers Gemma 2B, Gemma-2 2B and Phi-2, GPU only and without vision.

Bottom line: the bf16 fine-tune worked. Deploying multimodal Gemma 3n with a merged LoRA and intact vision was blocked by the tooling, as of my tests.

What did I end up using?

For Balùa I switched model and runtime: a 3B Qwen2.5-VL in GGUF with llama.cpp and the GPU through Vulkan. Gemma is still a good option if the official model is enough for you or your use case is text only. For a fine-tuned vision model, the GGUF route kept the vision working.

With Qwen, loading plus the first photo on the Pixel dropped to about 3 minutes, versus around 10 on the CPU, and later photos to about 2 min 30 s. The model takes 1.8 GB plus 1.3 GB for the vision projector.

Back in June I also trained Gemma 4 E2B, but MediaPipe didn’t support it then. Today Google already publishes Gemma 4 as .litertlm and LiteRT-LM runs it on Android. That’s the next thing I want to try.

If you’re still deciding whether running a model on the device is worth it, what size to pick or which quantization to use, start with the on-device AI guide. The full recipes, with the exact errors, are in the On-Device AI Cookbook.