# qwen-image-p150: Python API Use Qwen-Image-2.1 text-to-image on one Tenstorrent Blackhole p150 from Python code: scripts, pipelines and notebooks. The package wraps the port in `models/experimental/qwen_image_2_1`. It is the same pipeline that the HTTP server of the container uses, with the same defaults, so a call gives the same pixels as `POST /predict`. ## Install You need an environment that already has tt-metal (`ttnn`), built from tt-metal commit `8b98410e730b` with the patch in `patches/` of this repository applied (it adds dispatch on the chip's idle Ethernet cores). The default device configuration is ETH dispatch, 1 command queue and a 12x10 worker grid; the model uses only command queue 0. Without ETH dispatch support the package falls back to Tensix dispatch and gives a `RuntimeWarning`: a p150 then has an 11x10 worker grid and each step is slower. ```bash git clone https://huggingface.co/changh95/qwen-image-2.1-p150 export PYTHONPATH=$PWD/qwen-image-2.1-p150/code:$PYTHONPATH # the package and its bundled models/ tree pip install "numpy<2" safetensors "transformers==5.12.1" huggingface_hub loguru pillow # host-side dependencies ``` `code/` is used from the checkout (it is not a pip package). It does not install or change ttnn. The HTTP server (`models/experimental/qwen_image_2_1/server/app.py`) also needs fastapi, uvicorn and pydantic; the tests need pytest. ```bash cd qwen-image-2.1-p150 TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py -q # host tests only: no device python -m pytest code/tests/test_api_device.py -s # device tests: one chip ``` `TT_VISIBLE_DEVICES=none` makes sure that the host tests cannot open a device. They do not need the weights. ## Quickstart ```python from qwen_image_p150 import QwenImage with QwenImage.from_pretrained(device_id=0) as model: out = model("White furry llama with black sunglasses, smiling and happy, jumping", seed=42, num_steps=40) out.image.save("llama.png") # PIL.Image, 1024x1024 RGB print(out.timings) # seconds per stage ``` The first `from_pretrained` on a machine downloads the weights (`Qwen/Qwen-Image-2.1` at revision `790c926`, about 33 GB) into the Hugging Face cache and compiles the kernels, which takes many minutes. Later loads read the weights, pre-compile the programs for all prompt lengths and run the warm-up (see [Warm-up](#warm-up)), which makes the first call as fast as the later calls. Speed: see "Expected performance" on the model card; `code/scripts/bench_inproc.py` measures it. The console stays quiet by default (`verbose=False`): tt-metal info and warning lines and the load progress lines are not shown. Errors and Python warnings still show. Use `from_pretrained(verbose=True)` to see all log lines. `code/examples/quickstart.py` runs this snippet and writes `llama.png` and `llama.json` (the timings). ## More examples ```python import numpy as np from qwen_image_p150 import QwenImage model = QwenImage.from_pretrained() # same as device_id=0; close() when done # several prompts: one image each, in order (each result is the same as a single call) outs = model(["a red bicycle by a stone wall", "a lighthouse at dusk"], seed=[1, 2]) arr = outs[0].numpy() # uint8 [1024, 1024, 3] rgba = outs[0].numpy(rgba=True) # uint8 [1024, 1024, 4] (the model generates alpha) # diffusers-style names also work img = model("a lighthouse at dusk", num_inference_steps=40).images[0] model.close() ``` Image editing (condition images, `editing=True`, `images=...`) is not supported in this package: the code path exists, but it is not tested on this tt-metal build, and with the editing encoders loaded the chip hangs under ETH dispatch. Use an existing device (the object then does not close it): ```python import ttnn from qwen_image_p150 import QwenImage dev = ttnn.open_mesh_device(ttnn.MeshShape(1, 1), l1_small_size=32768, trace_region_size=640 << 20, dispatch_core_config=ttnn.DispatchCoreConfig(ttnn.DispatchCoreType.ETH)) model = QwenImage.from_pretrained(device=dev) ... model.close() ttnn.close_mesh_device(dev) ``` ## API reference ### `QwenImage.from_pretrained(weights=None, *, device_id=0, device=None, dispatch="auto", editing=False, size=1024, revision=None, dit_dtype="bf16", te_dtype="bfp8", warmup=True, warmup_variants=None, verbose=False, download=True, trace_region_size=None)` Opens the device, loads the text encoder (Qwen3-VL-8B language model), the 7B DiT and the VAE decoder onto the chip, compiles every program and captures the metal traces. Returns a `QwenImage`. | argument | meaning | |---|---| | `weights` | `None`: `Qwen/Qwen-Image-2.1` at the validated revision, from the HF cache (downloaded when missing; `QWEN_IMAGE_SNAPSHOT=` overrides). Can also be a local snapshot directory or another HF repo id with the same layout. | | `device_id` | Chip index among the visible devices (`TT_VISIBLE_DEVICES`). | | `device` | An open 1x1 `ttnn` mesh device to use. It needs `trace_region_size >= 640 MiB` and `l1_small_size=32768`. `close()` does not close it. | | `dispatch` | `"auto"`: ETH dispatch, so all 120 Tensix cores compute, with the worker grid capped at 12x10, as served. Falls back to Tensix dispatch with a `RuntimeWarning` if this tt-metal cannot open ETH dispatch. Uses Tensix dispatch when `editing=True`. Other values: `"eth"`, `"tensix"` (explicit opt-in: 11x10 on a p150). Always 1 command queue. `QWEN_IMAGE_ETH_DISPATCH=0/1` overrides `"auto"`. | | `editing` | Also loads the vision tower and the VAE encoder so that calls accept `images`. Not supported in this package (see above). | | `size` | Output side for text-to-image. Only 1024 is validated and has tuned matmul blockings. | | `dit_dtype`, `te_dtype` | Weight formats: `"bf16"` / `"bfp8"`. The defaults (`bf16` DiT, `bfp8` text encoder) are the published precision policy. | | `warmup` | Prepares the request variants of `warmup_variants` during the load, so that the first real call is as fast as later calls. `False`: no warm-up (the first call is then slower). See [Warm-up](#warm-up). | | `warmup_variants` | List of variant dicts for `model.warmup(**variant)`. `None` (default): see [Warm-up](#warm-up). `[]` is the same as `warmup=False`. | | `verbose` | `False` (default): a quiet console. tt-metal info and warning lines and ttnn debug lines are hidden (`TT_LOGGER_LEVEL=error`, `LOGURU_LEVEL=WARNING`, set only when you did not set them, and only before ttnn is first imported). Errors and Python warnings still show. `True`: all log lines. | | `download` | Set to `False` to fail instead of downloading missing weights. | ### `model(prompt, *, seed=42, num_steps=40, images=None, num_inference_steps=None, return_latents=False)` | argument | meaning | |---|---| | `prompt` | `str`. A list of `str` returns a list (see `generate_many`). Prompts up to 256 tokens use the pre-compiled path. The 4 most recent prompts stay on the device, so a repeated prompt skips text encoding and the prefix pass. Longer prompts work, but use a slower path. | | `seed` | `int`: seed of the initial noise (CPU `torch.randn` in bf16, the same noise as diffusers). For a list of prompts, one `int` for all prompts or a list with one seed per prompt. | | `num_steps` | Euler steps, 1..100 (40 is validated). `num_inference_steps` is an alias. | | `images` | Condition images for editing (not supported in this package; one image or a list, max 4). Each image can be a `PIL.Image`, a file path, encoded PNG/JPEG `bytes`, a `numpy.ndarray`, or a `torch.Tensor`. Arrays and tensors: HxW, HxWx3, HxWx4 or CxHxW, `uint8` or float in [0, 1]. All images are converted to RGBA, as the server does. Each image is resized to about 1 MP. The output has the aspect ratio of the last image. | | `return_latents` | Also returns the final latents (one more device read). | Returns `QwenImageOutput`: | field | type | |---|---| | `image` | `PIL.Image` RGB, composited over white. This is the server's `image`. 1024x1024 for text-to-image. | | `image_rgba` | `PIL.Image` RGBA, the 4-channel VAE output. | | `images` | `[image]`, the diffusers convention. | | `numpy(rgba=False)` | `uint8` array `[H, W, 3]` or `[H, W, 4]`. | | `save(path, rgba=False)` | Writes the image to a file. | | `size` | `(width, height)`. | | `prompt`, `seed`, `num_steps` | The request. | | `timings` | Seconds: `encode_s`, `prefix_s` (0 when the prompt is resident), `denoise_s` (40 steps, with the VAE decode in the same trace), `decode_s` (image download + PIL), `total_s`, `call_s`. | | `latents` | `torch.Tensor [1, H/16*W/16, 64]` bf16 (only with `return_latents=True`). | ### `model.warmup(num_steps=40, *, return_latents=False, images=None)` Prepares one request variant, so that its first real call is as fast as later calls. A variant that is already prepared costs nothing (idempotent). Outputs do not change. Returns a dict of the work done and its time in seconds. | argument | meaning | |---|---| | `num_steps` | A step count or a list. The first text-to-image warm-up runs one full generation of a dummy prompt at the first step count. Each listed step count also gets its host-side schedule prepared. | | `return_latents` | Also runs the latents readback once. | | `images` | Editing only (not supported in this package): the number (1..4) of 1:1 condition images for one dummy edit. | ### `model.generate_many(prompts, *, seed=42, num_steps=40, images=None, return_latents=False)` Generates one image per prompt, in order, with batch 1 on the device. Each result is identical to a single call with the same prompt and seed. Returns a list of `QwenImageOutput`. ### Lifetime and other members - `model.close()` releases all metal traces and closes the device if `from_pretrained` opened it. You can call it more than once. `with QwenImage.from_pretrained() as model:` calls it at the end of the block. At interpreter exit, a model that is still open is closed. - One object owns one chip. Calls are serialised with a lock, so you can share one object between threads. - `model.info`: weights path, worker grid, dispatch, load time, warm-up time and the prepared variants (`warmed`). - `model.pipeline` and `model.device`: the port's `QwenImage21Pipeline` and the ttnn device, for advanced use. - `qwen_image_p150.load_image(x)`: the image conversion that the API uses (gives PIL RGBA). ## Warm-up `from_pretrained` prepares the request variants that most users need, so that the first real call is as fast as the later calls. The warm-up does not change outputs. It does not keep results: the warm-up prompt is removed from the prompt cache after the warm-up, so no real call can get a cached result from it. ### What changes the work of a call | request parameter | what changes on the device / host | prepared by the default warm-up? | |---|---|---| | prompt length 1..256 tokens | text-encoder and attention shapes per 32-token bucket | yes: compiled for all lengths at load time (`QWEN_PREWARM=1`), with or without warm-up | | new prompt text | text encoder, prefix pass, one step-trace capture | every new prompt does this work; it is part of the steady-state time, not a first-call cost | | first request of the process | tokenizer load, step-trace and VAE-trace capture, host buffers | yes: one 40-step generation of a dummy prompt | | `num_steps` | the host-side schedule (modulation rows) is computed once per step count and uploaded | yes for 40, 50, 30 and 20; other counts compute it on their first call | | `return_latents=True` | one more device read | yes (the dummy generation reads the latents) | | `seed`, list of prompts | nothing new | not needed | | prompt longer than 256 tokens | variable-shape path: new programs for each new length, release of all traces | no (see below) | ### Defaults and options ```python model = QwenImage.from_pretrained() # default warm-up model = QwenImage.from_pretrained(warmup_variants=[{"num_steps": [40, 25, 60]}]) # other step counts model = QwenImage.from_pretrained(warmup=False) # fastest load, slow first call model.warmup(num_steps=8) # prepare one more variant later model.warmup(num_steps=8) # again: no work (idempotent) print(model.info["warmed"], model.info["warmup_s"]) ``` The default is `warmup_variants=[{"num_steps": [40, 50, 30, 20], "return_latents": True}]`, plus `{"images": 1}` first when `editing=True`. The default text-to-image warm-up is one 40-step generation, the first trace captures and the four host schedules. Host memory: about 80 KB per prepared step, 11 MB for the default. Device memory: no extra DRAM and no extra trace (the warm-up uses the same step trace and VAE trace as any call; its prompt slot is freed afterwards). ### Variants that cannot be prepared - **Prompts longer than 256 tokens.** They use a variable-shape path: each new length compiles new programs (text encoder per 32-token bucket, prefix and attention per exact length), releases all traces and captures a 40-step loop trace, so every such call is slower than a call with a short prompt, on the first call and on later calls; their speed is not measured. The length is not bounded (2000 characters can be more than 2000 tokens), so it cannot be compiled in advance. Keep prompts at 256 tokens or fewer. - **Step counts that are not prepared** compute their host-side schedule on their first call. Use `warmup_variants` or `model.warmup(num_steps=...)` for the step counts that you use. ## Notes - Only batch 1 and no classifier-free guidance. This is how the model is sampled. - The package contains the port's `models/` tree (`models.experimental.qwen_image_2_1` and modified copies of some `models.tt_dit` / `models.common` files from tt-metal). When you import `qwen_image_p150`, these copies are put first in the `models` namespace, also when `$TT_METAL_HOME` is on `PYTHONPATH`. Import `qwen_image_p150` before other tt-metal `models.*` modules in the same process. - The precision and performance settings (`QWEN_*` environment variables, listed under `container.serve.env` in `tt_kernel_manifest.json`) keep their served defaults (the container sets them explicitly). Do not change them unless you want to experiment. - Tests: `TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py` (no device; needs pytest) and `python -m pytest code/tests/test_api_device.py -s` (one chip, weights in the HF cache). The device test starts with `test_first_call_fast`: after the default warm-up, the first real call (40 steps, and 20 steps) must be within 10 % of the later calls. `code/tests/bench_first_call.py` measures first-call latency per variant in a fresh process.