How to use from the
Use from the
LiteRT-LM library
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM)
# and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter).
# For platform-specific integration guides, please refer to the official developer website:
# https://ai.google.dev/edge/litert-lm

# To try LiteRT-LM, the easiest way is to use our CLI tool.
# 1. Install the LiteRT-LM CLI tool:
pip install -U litert-lm

# 2. Download and run this model locally:
# See: https://ai.google.dev/edge/litert-lm/cli
# A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run
# (or pass its name right after the repo id).
litert-lm run \
  --from-huggingface-repo=litert-community/InternVL3_5-2B \
  --prompt="Write me a poem"

InternVL3.5-2B β€” LiteRT-LM (on-device Vision-Language Model)

OpenGVLab/InternVL3_5-2B converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the same runtime that runs litert-community/FastVLM-0.5B). On 2026-11-15 model.litertlm was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. Outside the dated entries under Conversion notes, "this file" means that file and "the previous file" the one it replaced. On a Galaxy S26 GPU this file decodes 40.2 tokens/s and prefills 1,024 tokens at 537 tokens/s (LiteRT-LM v0.19.0).

InternVL3.5-2B is a compact vision-language model: an InternViT vision encoder + pixel-shuffle + MLP projector feeding a Qwen3-1.7B language decoder (the newer Qwen3 backbone is what distinguishes it from the InternVL3-2B build, which used Qwen2.5-1.5B). This bundle runs it through LiteRT-LM's fast_vlm multimodal path β€” give it an image and a question, get a grounded answer, fully on-device.

File model.litertlm (~1.62 GB)
Vision InternViT encoder + pixel-shuffle + MLP projector, int8 weights β€” single 448Γ—448 image β†’ 256 image tokens
Decoder Qwen3-1.7B, int4 weights (symmetric, blockwise-32 + OCTAV optimal-clipping); input embedding INT8 (externalized section)
Compute integer
Context (KV cache) 2048
Image input resized to 448Γ—448 (ImageNet normalization is baked into the vision encoder)
Base model OpenGVLab/InternVL3_5-2B (Apache-2.0)

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command β€” this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/InternVL3_5-2B model.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this picture in one sentence." --top-k 1

On the COCO photo 000000039769.jpg (640 Γ— 480, two tabby cats on a pink blanket), this file answers:

Two tabby cats are sleeping on a pink couch next to two remote controls.

This answer came from litert-lm 0.19.2 on a Mac (2026-11-15), with the file given by its path (the copy that was uploaded to this repo, with the same sha256). Two runs of this file with the flags above gave this answer and "Two tabby cats are sleeping on a pink couch next to two remotes."; one run of the previous file (downloaded from this repo and given by its path) gave "Two tabby cats are sleeping on a pink couch next to remote controls." A third run of this file, with the command exactly as above (it downloaded the file from this repo), gave that last sentence too. With --backend cpu --vision-backend cpu --cpu-thread-count 1 --cache no --max-num-tokens 2048 added, litert-lm 0.19.0, 0.19.1 and 0.19.2 gave the answer above with both files (2026-11-14).

Leave --vision-backend unset, as above: this file, like the previous one, sets no default backend, so with --backend and --vision-backend both unset, litert-lm 0.19.0 to 0.19.2 run the decoder and the vision encoder on the CPU (unless a backend is set for the model in the CLI's config). With --backend gpu added and --vision-backend still unset, this file answered the picture in one run (on a Mac with litert-lm 0.19.2: "Two tabby cats are sleeping on a pink couch, with remote controls placed beside them."). With --backend gpu --vision-backend gpu, neither this file nor the previous file answers a picture on a Mac (litert-lm 0.19.0 to 0.19.2; the reply is An error occurred); with litert-lm 0.19.2 the error log names two op types of the vision encoder that the GPU delegate does not take, BROADCAST_TO and GATHER_ND, and the runtime does not run the encoder split between the GPU and the CPU. On a Galaxy S26 and a Pixel 11 Pro (LiteRT-LM v0.19.0), litert_lm_advanced_main with the vision encoder on the GPU stops too (see the Galaxy S26 section).

If run stops right after the download with no error message from the CLI (a bus error; exit status 138 on a Mac) or with an error log that contains too large for file size, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in ~/.litert-lm/cache/huggingface/litert-community/InternVL3_5-2B/ and uses it again on later runs without checking its size. Delete that file and run the command again.

If you downloaded model.litertlm with --from-huggingface-repo before it was replaced on 2026-11-15 (02:46 UTC), delete ~/.litert-lm/cache/huggingface/litert-community/InternVL3_5-2B/model.litertlm, then run the command again. Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images (but see the Known limitation below). litert-lm serve exposes this bundle as a local OpenAI-compatible API. None of these was tried with this file, and on desktop this file was run on macOS only.

Performance

Measured on an earlier file: litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device Backend Prefill (256) Decode TTFT
Apple M4 Max (macOS) CPU 138 tok/s 31.9 tok/s 2.04 s
Apple M4 Max (macOS) GPU (Metal) 1831 tok/s 137.2 tok/s 0.15 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output. With litert-lm 0.19.1 and 0.19.2, this file and the previous one answer a text question on the Mac GPU with the same reply, byte for byte (with 0.19.0 the previous file's reply differs from its 30th line on, and this file's reply is the same on all three versions); with litert-lm 0.19.2 and --backend gpu --vision-backend cpu, both answer the picture under How to use with "Two tabby cats are sleeping on a pink couch, with remote controls placed beside them."

Accuracy note

The vision tower converts faithfully to the reference β€” float CPU-parity end-to-end corr β‰ˆ 1.0 (max abs diff ~1e-4), with no FLEX/CUSTOM fallback ops; int8 vision weights preserve grounding. The Qwen3-1.7B decoder uses the same blockwise-32 + OCTAV int4 recipe that scores 90.7% GSM8K on the sibling Ministral-3-3B-Reasoning build. On a reference eager run the model describes photos accurately and in detail (e.g. a black-and-white Ansel-Adams-style landscape β†’ "dramatic mountain landscape … snow-capped peaks … a winding river through a forested valley").

Galaxy S26 β€” GPU and CPU

Each run prefills 1,024 tokens and generates 256 tokens, three times (three iterations): the runtime's benchmark mode puts the prompt file's one sentence ("Explain what on-device AI means in simple terms.") in the chat template as a user message and pads the tokens to 1,024 with token id 0. Iteration 1 is the first pass after the engine starts. The GPU row is the median of iterations 2–3 over two runs, so it leaves iteration 1 out; on the GPU, this file's iteration 1 prefilled faster and decoded more slowly than its iterations 2 and 3 (iteration 1: prefill 580–591 tok/s, decode 39.0–39.3 tok/s). On the CPU the median prefill and decode speeds drop after iteration 1, so each iteration has its own row: the median of that iteration over two runs.

Backend Prefill, 1,024 tokens Decode Time to first token Peak memory
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2–3 537 tok/s 40.2 tok/s 1.93 s 0.89 GB + 2.47 GB GPU driver
CPU (XNNPACK for most nodes), iteration 1 211 tok/s 19.3 tok/s 4.91 s 2.34 GB
CPU (XNNPACK for most nodes), iteration 2 104 tok/s 16.0 tok/s 9.93 s 2.34 GB
CPU (XNNPACK for most nodes), iteration 3 104 tok/s 16.0 tok/s 9.94 s 2.34 GB

Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16), litert_lm_advanced_main built from the LiteRT-LM v0.19.0 source (commit 771b06e1; the v0.19.0 tag is on the next commit, which changes only Package.swift and README.md), --benchmark with 1,024 prefill tokens and 256 decode tokens, context 2,048, 2026-11-15. Each run is a new process. The rows cover the text path. The GPU runs started at Android's thermal status 0 (none) and each ended at thermal status 1 or 2 (light or moderate), with the GPU's highest allowed clock one or two steps down; in every GPU run the prefill slowed from one iteration to the next. The CPU runs started at thermal status 1 and were at thermal status 2 (moderate) by 11–23 s in (the status is read about every 11 s), where they stayed until they ended.

In each of the four counted CPU runs (two per file, after one warm-up run that is not counted), the phone lowers the clock caps of its two CPU clock domains from 2–6 s in, to 1.56 / 1.50 GHz by 20–22 s in. Iteration 1 runs while the caps come down, iterations 2 and 3 at the lowered caps.

Peak memory is per run, so the three CPU rows share one value. It is the process's VmHWM: on the CPU 2.33–2.34 GB in this file's two counted runs (2.36 GB in the warm-up run) and 2.16 GB in each of the previous file's two. VmHWM does not count memory the GPU driver holds, so the GPU row adds the rise of the phone's GPU allocation counter (kgsl_page_alloc) during each of this file's GPU runs (2.47 GB; 1.85 GB in each of the previous file's). In the CPU runs that counter rose by less than 1 MB (0.44 MB in one of the five runs, not at all in the other four).

With the vision encoder on the GPU (--vision_backend=gpu), litert_lm_advanced_main stops with exit status 13 before it reads the picture: with this file on this phone, and with this file and the previous one on a Pixel 11 Pro. Its log names two op types of the encoder that the GPU delegate does not take, BROADCAST_TO and GATHER_ND (on this phone, 13 of the encoder's 1,320 nodes). With the vision encoder on the CPU (--vision_backend=cpu) and the decoder on the GPU, this phone answered "Describe this picture in one sentence." about the photo under How to use with "Two tabby cats are sleeping on a pink couch, with remote controls placed beside them." (one run).

On the GPU this file decodes about 37 % faster than the previous file. Four runs went previous, this, this, previous (each run's mean of iterations 2–3): prefill 558, 537, 542 and 539 tok/s, decode 29.4, 40.2, 40.3 and 29.4 tok/s; from the previous file to this one, decode Γ—1.366 and prefill Γ—0.962 in the first two runs, decode Γ—1.371 and prefill Γ—1.005 in the last two, while the previous file's own two runs differ by 3.6 % in prefill. On the CPU, this file's medians in iteration 1 were 1.6 % below the previous file's in prefill and 2.6 % above in decode, and in iterations 2 and 3 between 0.3 % and 0.9 % above; two counted runs of the same file differed by up to 1.6 %. The runs behind each number, and the S26 section this card carried for an earlier file: MEASUREMENTS.md.

On the Galaxy S26 the GPU decodes 2.1Γ— as fast as the CPU's iteration 1 and 2.5Γ— as fast as its iterations 2 and 3, and prefills 2.5Γ— and 5.2Γ— as fast, so this repo's litertlm_manifest.json recommends gpu for flagship-class Android. That recommendation rests on the Galaxy S26 alone.

GPU wiring, including the Gallery import toggle: GPU guide.

Pixel 11 Pro β€” GPU

On a Pixel 11 Pro GPU (the same build and workload, 2026-11-14) this file decodes 16.9 tok/s, 26 % faster than the previous file, and prefills 533 tok/s, 5 % faster (the median of iterations 2–3 over two runs per file). The Pixel 11 Pro CPU was not measured, so whether the GPU or the CPU is faster on that phone is not known. The runs: MEASUREMENTS.md.

⚠️ Known limitation β€” one image per conversation on the GPU (Metal) backend

Single-image VQA β€” the primary use case β€” worked in each setup tried with the decoder on the GPU and the vision encoder not on the GPU (one run each: on a Mac with litert-lm 0.19.2, --backend gpu with --vision-backend unset or cpu; on a Galaxy S26, --vision_backend=cpu; see How to use and the Galaxy S26 section). But on the GPU (Metal) backend of Apple devices, a second image in the same conversation truncates the answer β€” ask about one image per chat (start a new conversation for a different image). This is GPU-delegate-specific, not a model/bundle issue: on the CPU backend, multi-image works. The same GPU truncation reproduces with Apple's litert-community/FastVLM-0.5B, so it is not specific to this model. For reliable multi-image, run on the CPU backend. This was seen with an earlier file and was not tried with this file or on Android.

Run on iPhone / macOS

Use swift-litert-lm, a community Swift package over the LiteRT-LM runtime, and its LiteRTDemo sample. Load model.litertlm with the image (vision) tower enabled (modalities [.vision]), attach a photo, and ask a question.

Note for app integrators: this is a vision-only bundle (no audio tower). Bring up the engine with the vision modality only (Modality.textImage / [.vision]) β€” requesting the audio tower (.all) on a bundle with no audio section fails at session creation.

These steps were written for an earlier file and were not run with this one. On a Mac, this file does not answer a picture with the vision encoder on the GPU (--vision-backend gpu; see How to use).

Run on Android β€” Google AI Edge Gallery

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download model.litertlm, import it (tap +), attach an image and ask. The bundle already carries the tokenizer and prompt template.

These steps were written for an earlier file and were not repeated with this one, and which backend Gallery gives the vision encoder was not checked. On a Galaxy S26 and a Pixel 11 Pro (LiteRT-LM v0.19.0), litert_lm_advanced_main with the vision encoder on the GPU stops with exit status 13; with the vision encoder on the CPU, the Galaxy S26 answers the picture (see the Galaxy S26 section).

Conversion notes

  • LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,448,448,3]β†’[1,256,4096]) + VISION_ADAPTER ([1,256,4096]β†’[1,256,2048], matched to the Qwen3-1.7B hidden size) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
  • The vision encoder bakes InternVL's ImageNet normalization and the NCHW transpose into the graph (the runtime feeds a [0,1] NHWC image).
  • The InternViT attention is rewritten 4D-clean (qkv split before the head reshape, avoiding a 5D intermediate) for the GPU delegate.
  • Decoder extracted from the InternVLChat wrapper as a standalone Qwen3ForCausalLM (dynamic rope_scaling stripped; exported with cache ≀ base max so base RoPE is exact).

Each dated entry below was written on its date: in it, "this file" is the file published that day, "the previous file" is the one it replaced, and "this card" is the card as it was then.

2026-08-28 β€” start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token β€” so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file β€” only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-30 β€” tokenizer section replaced (weights unchanged)

The tokenizer in model.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (Γ©, Γ±, ΓΌ, Β°, Β·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and 36 of the 223 standalone characters U+00A1–U+017F in the checks (Β‘, Δ€ to Δ‘, and Ε²) became the token the conversion had reused as UNK β€” the end-of-text token for this vocabulary. model.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token β€” only the file's own sha256 differs. Verified on the LiteRT-LM runtime: one user message ("What is the capital of France?") as the chat template renders it, 7 test strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions gave byte-identical answers to the previous file within the check's limit of 48 output tokens (same ids in, same tokens out). In the checks, text with accented letters, symbols or non-Latin scripts and a short Python snippet reach the model differently from before (digits, plain English text and the test emoji do not), so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too β€” the on-device check has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 β€” chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)

The chat template in model.litertlm matched history turns only on role 'model', so a conversation created through the Conversation API with the standard role 'assistant' (for example create_conversation(messages=[...]) restoring a chat) silently dropped every assistant turn from the rendered prompt β€” the model saw consecutive user turns with its own replies gone. The template's role conditions now also accept 'assistant'. The engine's own send_message flow was never affected.

Template-only change: every section of the bundle except the metadata is byte-identical to the previous file (verified by per-section sha256), and single-turn renders, token ids, prefill counts and greedy answers are unchanged (verified on the LiteRT-LM runtime), so the numbers on this card still describe this file β€” only the file's own sha256 differs. Re-download only matters if you inject conversation history with role 'assistant'. Details: google-ai-edge/LiteRT-LM#3417 (https://github.com/google-ai-edge/LiteRT-LM/issues/3417).

2026-11-15 β€” GPU graph and a 1,024-token prefill (weights unchanged)

The prefill/decode graph now has the GPU form litert-torch writes with --apply_gpu_composites, plus a 1,024-token prefill signature, prefill_1024: a copy of prefill_512 at a length of 1,024 tokens that shares its weight buffers. Every weight, and every section outside that graph, is byte-identical to the previous file. The whole file went from sha256 acc9a204… (1,613,250,560 B) to 1d36c24c… (1,615,527,936 B). On a Mac the two files give the same replies byte for byte: 4 prompts and a picture on the CPU (litert-lm 0.19.2, 1 thread, greedy decoding with top-k 1), and one prompt and the picture on the CPU with litert-lm 0.19.0 to 0.19.2. On the Mac GPU the two files give the same reply to that prompt with litert-lm 0.19.1 and 0.19.2; with 0.19.0 the previous file's reply differs from its 30th line on, and this file's reply is the same on all three versions. 8 short questions: 7 of 8 with both files on the Mac CPU and GPU (both answer "cool" as the opposite of "hot"); on a Pixel 11 Pro GPU 7 of 8 with the previous file and 8 of 8 with this file, which answers "cold" there; 8 of 8 with this file on a Galaxy S26 GPU (the previous file was not asked there). Galaxy S26 GPU (LiteRT-LM v0.19.0, four runs in the order previous, new, new, previous; each run's mean of iterations 2–3): prefill 558, 537, 542 and 539 tokens/s, decode 29.4, 40.2, 40.3 and 29.4 tokens/s, so from the previous file to this one prefill Γ—0.962 and decode Γ—1.366 in the first pair, prefill Γ—1.005 and decode Γ—1.371 in the second; the previous file's two runs differ by 3.6 % in prefill. Procedure: REPRODUCE.md. Measured on this file: the last sentence of the paragraph under the title, the file size, the example answer under How to use and the two paragraphs after it, the last sentence of the paragraph under the Performance table, the Galaxy S26 and Pixel 11 Pro sections, the first sentence of the Known limitation section, the last paragraph of the iPhone / macOS section and of the Gallery section, and this entry. The Known runtime issues below come from earlier runtimes or other files (each says which) and were not re-checked with this file. The rest of this card was written for earlier files; the weights, tokenizer and chat template it describes are the same in this file.

Known runtime issues

External weights (since 2026-11-09). In model.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The last file with the weights inside the graph stays at revision b62b8ba.

From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error. This was measured with other models' files, not with this one.

Downloads last month
390
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for litert-community/InternVL3_5-2B

Collection including litert-community/InternVL3_5-2B