---
title: "Gemma on Android: running an on-device LLM with LiteRT"
description: "How I got Gemma running on an Android phone: exporting the model to .litertlm, wiring it into the app and what performance to expect, mistakes included."
author: "Marc Mayol"
lang: "en"
published: "2026-10-28"
canonical: "https://marcmayol.com/blog/en/gemma-on-android-on-device-llm/"
translation: "https://marcmayol.com/blog/es/gemma-en-android-llm-on-device/"
---

# Gemma on Android: running an on-device LLM with LiteRT

![Gemma on Android: running an on-device LLM with LiteRT](https://marcmayol.com/gemma-on-android.png)

This post is the hands-on half of my [guide to running an LLM locally](https://marcmayol.com/blog/en/run-llm-locally-on-device-ai-guide/). Here I zoom in on one concrete case: getting Gemma onto an Android phone so it works offline and your data never leaves the device. If you wonder why that last bit matters so much, I covered it in my post on [Shadow AI](https://marcmayol.com/blog/en/shadow_ai/).

I did it for **Balùa**, my app that looks at a photo of a plate and tells you which foods are on it. I fine-tuned Gemma to return JSON with the foods, exported it to LiteRT's format and ran it on a Pixel 8 Pro. It worked, with caveats. Here are the steps, the errors and how long it took.

When I looked for how to do this, most of what I found skipped the failures. So here they are.

## What do you need to run Gemma on Android?

You need three things: a Gemma model in `.litertlm` format, a runtime in the app to load it (LiteRT-LM, or MediaPipe LLM Inference, which is what I used) and an arm64 Android phone with at least 6 GB of RAM. If you won't fine-tune, download Google's ready-made conversion and skip the hard part.

The pieces, in order:

- **The model.** Google publishes converted Gemma models on Hugging Face under [litert-community](https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm). The Gemma 4 E2B one is about 2.6 GB.
- **The runtime.** I used MediaPipe `tasks-genai` (versions 0.10.29 to 0.10.35). Today Google is pushing [LiteRT-LM](https://developers.google.com/edge/litert-lm/android), with a Kotlin API and CPU, GPU and NPU support.
- **The device.** arm64, 6 GB of RAM or more, and room for the model, which doesn't go inside the APK.

My advice: if the official model does the job, use it. The official int4 Gemma 3n E2B identified food on my Pixel out of the box. The next section is only needed if you've trained your own version.

## How do I export Gemma to .litertlm?

You export it with `litert-torch`, which only ships builds for Linux and macOS. On Windows, use WSL. The keys are pinning `transformers` to 4.56.2, giving WSL plenty of swap, and having the SentencePiece tokenizer plus a `special_tokens_map.json` with `eoi_token` in the model folder. After that it's a single command.

The litert-torch issues (#994 and #1039) treated exporting Gemma 3n with vision as broken. It can be done. This is the environment that worked for me, exactly as it appears in the [cookbook](https://github.com/marcmayol/on-device-ai-cookbook):

```bash
uv venv exportenv --python 3.12
UV_LINK_MODE=copy uv pip install 'litert-torch==0.9.1' 'transformers==4.56.2' pillow torchvision timm
```

**The `transformers` version matters most.** With 5.x I had to patch litert-torch because the model's internal structure changed. The export finished without errors and the model came out broken, generating things like `df online df online...`. With 4.56.2 those patches aren't needed.

A `uv` detail: it uses hard links. If you reuse an environment, reinstall with `UV_LINK_MODE=copy` so you don't inherit patches from another one.

**Memory.** The step that builds the MLIR module for a ~11 GB model ran out of memory with 31 GB of RAM, and WSL killed it with a bare `Killed`. I fixed it by giving WSL 28 GB of memory and 72 GB of swap in `.wslconfig`, then restarting with `wsl --shutdown`.

**Two files that must be in the model folder:**

1. `tokenizer.model` (SentencePiece, about 4.7 MB). Gemma 3n's fast tokenizer only ships `tokenizer.json`, and then MediaPipe fails with `SentencePiece tokenizer is not found in the model`. Download it from `google/gemma-3n-E2B-it`.
2. `special_tokens_map.json` with `eoi_token`. If it's missing, the export dies with `KeyError: 'eoi_token'`. Copy it from the base model.

And the command:

```bash
litert-torch export_hf /ruta/merged_gemma3n /out \
  --task=image_text_to_text --export_vision_encoder \
  --quantization_recipe=dynamic_wi4_afp32
```

It takes about 30 minutes and produces a `model.litertlm` of about 2.7 GB with vision. Watch the quantization recipe names, they're not intuitive:

| Recipe | What it does | When to use it |
|---|---|---|
| `dynamic_wi4_afp32` | 4-bit weights | First attempt, smallest size |
| `dynamic_wi8_afp32` | 8-bit weights, roughly 2x the size | If int4 degrades the output |
| `weight_only_wi4_afp32` / `wi8` | Quantizes weights only | Alternative to the dynamic ones |
| `static_wi8_ai8` / `ai16` | Weights and activations | When you need to squeeze more |

`dynamic_int4` doesn't exist, even though it's the first thing you'll type.

## How do I wire the model into the app?

Copy the `.litertlm` to the app's external files folder, never to `/data/local/tmp`, and load it with the runtime from a background thread. With MediaPipe the GPU backend crashed my app, so I forced CPU. Keep `maxTokens` at 1024 or more, because the image alone takes about 258 tokens.

The model doesn't fit in the APK. During development I push it with `adb push` to `Android/data/<package>/files/`, the folder `getExternalFilesDir` returns. The app reads it without special permissions. From `/data/local/tmp` the app can't read it and fails without a clear message.

The MediaPipe configuration I used, from the cookbook:

```kotlin
LlmInference.LlmInferenceOptions.builder()
    .setModelPath(path)                       // en getExternalFilesDir, NO /data/local/tmp
    .setMaxTokens(1024)                        // input imagen ~258 tokens; no bajar
    .setPreferredBackend(LlmInference.Backend.CPU)   // el backend GPU CASCA (SIGSEGV)
    ...
```

Three things I ran into:

- **The GPU backend crashed** with a SIGSEGV while creating the OpenCL executor. On the CPU it was stable.
- **The model repeated the JSON in a loop.** I solved it by parsing with a regex and dropping duplicates instead of expecting clean JSON.
- **Android warned that the native library wasn't 16 KB aligned.** On a device with 4 KB pages (check with `getconf PAGESIZE`) it's only a warning and the model loads anyway.

If you're starting today, look at LiteRT-LM before MediaPipe. According to the [official docs](https://developers.google.com/edge/litert-lm/android), you set up an `EngineConfig` with the model path and the backend (with a separate one for vision), initialise the `Engine` off the main thread and talk to it through `createConversation()`. I haven't tried it in Balùa yet, so I can't tell you about its failure modes.

A design tip: put the analyser behind an interface. My app had an analyser that fell back to a mock when it couldn't find the model. That let me build the UI without depending on the model, and later switch models without touching anything else. It's what I describe in [hexagonal architecture and LLMs](https://marcmayol.com/blog/en/hexagonal-architecture-and-llms/).

## What performance can I expect?

It depends on the task. For text, Google reports around 47 tokens per second on CPU and 52 on GPU for Gemma 4 E2B on a Galaxy S26 Ultra. For a photo with Gemma 3n on the CPU of my Pixel 8 Pro, each analysis took several minutes, with the CPU at 365-421% and around 3 GB of RAM.

These are Google's numbers for Gemma 4 E2B with LiteRT-LM, taken from its [Hugging Face model card](https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm) (1,024 input tokens, 256 output tokens):

| Device | Backend | Prefill (tok/s) | Decode (tok/s) | First token | Memory |
|---|---|---|---|---|---|
| Galaxy S26 Ultra | CPU | 557 | 46.9 | 1.8 s | 1,733 MB |
| Galaxy S26 Ultra | GPU | 3,808 | 52.1 | 0.3 s | 676 MB |
| My Pixel 8 Pro, Gemma 3n with a photo | CPU (MediaPipe) | n/a | n/a | several minutes per photo | ~3 GB |

The last row is mine and isn't a fair comparison: different model, different runtime, an older phone and, above all, an image as input. Encoding the image is expensive. But it's what I actually saw, and it's a good reason not to promise instant answers in a multimodal app.

## What breaks and how do you catch it?

The conversion is what breaks most: the export finishes fine and the model comes out broken. You catch it by comparing against the bf16 model on the PC and against Google's official `.litertlm`, always with two different images. If two different photos get the same answer, the vision encoder is broken.

That was my case. With the export fixed, my fine-tuned Gemma 3n returned flawless JSON. But two different photos got the same list of foods, with the same portions. This is how I isolated it:

| Test | Result | What it told me |
|---|---|---|
| My model in bf16 on the PC | Perfect | Fine-tuning and prompt were fine |
| Google's official `.litertlm` on the Pixel | It could see: food → foods, portrait → "Man" | The runtime was fine |
| My exported model, int4 | Made-up foods, garbled words | Quantization was degrading it |
| My exported model, int8 | Clean JSON, but the same for any photo | Vision was lost in conversion |

There was no plan B inside MediaPipe either: its LoRA support only covers Gemma 2B, Gemma-2 2B and Phi-2, GPU only and without vision.

Bottom line: the bf16 fine-tune worked. Deploying multimodal Gemma 3n with a merged LoRA and intact vision was blocked by the tooling, as of my tests.

## What did I end up using?

For Balùa I switched model and runtime: a 3B Qwen2.5-VL in GGUF with llama.cpp and the GPU through Vulkan. Gemma is still a good option if the official model is enough for you or your use case is text only. For a fine-tuned vision model, the GGUF route kept the vision working.

With Qwen, loading plus the first photo on the Pixel dropped to about 3 minutes, versus around 10 on the CPU, and later photos to about 2 min 30 s. The model takes 1.8 GB plus 1.3 GB for the vision projector.

Back in June I also trained Gemma 4 E2B, but MediaPipe didn't support it then. Today Google already publishes Gemma 4 as `.litertlm` and LiteRT-LM runs it on Android. That's the next thing I want to try.

If you're still deciding whether running a model on the device is worth it, what size to pick or which quantization to use, start with the [on-device AI guide](https://marcmayol.com/blog/en/run-llm-locally-on-device-ai-guide/). The full recipes, with the exact errors, are in the [On-Device AI Cookbook](https://github.com/marcmayol/on-device-ai-cookbook).

## FAQs

### What is a .litertlm file?

It's the model format used by LiteRT-LM, Google's runtime for running LLMs on device. It packs the quantized weights, the tokenizer and, for multimodal models, the vision encoder into a single file. Google publishes ready-made Gemma conversions on Hugging Face under the litert-community organisation.

### Can I export a Gemma model I fine-tuned to .litertlm?

Yes, with litert-torch. It worked for me in WSL with transformers 4.56.2 and the SentencePiece tokenizer in the model folder. But with multimodal Gemma 3n and a merged LoRA, the exported model stopped seeing images. If your use case is text only, or the base model is good enough, the path is much simpler.

### What phone do I need to run Gemma on-device?

An arm64 Android phone with 6 GB of RAM or more. Gemma 3n E2B with vision takes about 2.7 GB on disk and in my tests used around 3 GB of RAM on the CPU. I tested it on a Pixel 8 Pro.

### Is Gemma fast on a phone?

It depends a lot on the task and the backend. Google reports around 47 tokens per second of decoding on CPU and 52 on GPU for Gemma 4 E2B on a Galaxy S26 Ultra. In my case, with multimodal Gemma 3n analysing photos on the CPU because the GPU crashed, each photo took several minutes.
