changh95's picture
tt-model push qwen-image-2.1-p150 (container)
95c57e6 verified
|
Raw History Blame Contribute Delete
14.9 kB

qwen-image-p150: Python API

Use Qwen-Image-2.1 text-to-image on one Tenstorrent Blackhole p150 from Python code: scripts, pipelines and notebooks. The package wraps the port in models/experimental/qwen_image_2_1. It is the same pipeline that the HTTP server of the container uses, with the same defaults, so a call gives the same pixels as POST /predict.

Install

You need an environment that already has tt-metal (ttnn), built from tt-metal commit 8b98410e730b with the patch in patches/ of this repository applied (it adds dispatch on the chip's idle Ethernet cores). The default device configuration is ETH dispatch, 1 command queue and a 12x10 worker grid; the model uses only command queue 0. Without ETH dispatch support the package falls back to Tensix dispatch and gives a RuntimeWarning: a p150 then has an 11x10 worker grid and each step is slower.

git clone https://huggingface.co/changh95/qwen-image-2.1-p150
export PYTHONPATH=$PWD/qwen-image-2.1-p150/code:$PYTHONPATH    # the package and its bundled models/ tree
pip install "numpy<2" safetensors "transformers==5.12.1" huggingface_hub loguru pillow   # host-side dependencies

code/ is used from the checkout (it is not a pip package). It does not install or change ttnn. The HTTP server (models/experimental/qwen_image_2_1/server/app.py) also needs fastapi, uvicorn and pydantic; the tests need pytest.

cd qwen-image-2.1-p150
TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py -q    # host tests only: no device
python -m pytest code/tests/test_api_device.py -s                         # device tests: one chip

TT_VISIBLE_DEVICES=none makes sure that the host tests cannot open a device. They do not need the weights.

Quickstart

from qwen_image_p150 import QwenImage

with QwenImage.from_pretrained(device_id=0) as model:
    out = model("White furry llama with black sunglasses, smiling and happy, jumping", seed=42, num_steps=40)
    out.image.save("llama.png")      # PIL.Image, 1024x1024 RGB
    print(out.timings)               # seconds per stage

The first from_pretrained on a machine downloads the weights (Qwen/Qwen-Image-2.1 at revision 790c926, about 33 GB) into the Hugging Face cache and compiles the kernels, which takes many minutes. Later loads read the weights, pre-compile the programs for all prompt lengths and run the warm-up (see Warm-up), which makes the first call as fast as the later calls. Speed: see "Expected performance" on the model card; code/scripts/bench_inproc.py measures it.

The console stays quiet by default (verbose=False): tt-metal info and warning lines and the load progress lines are not shown. Errors and Python warnings still show. Use from_pretrained(verbose=True) to see all log lines.

code/examples/quickstart.py runs this snippet and writes llama.png and llama.json (the timings).

More examples

import numpy as np
from qwen_image_p150 import QwenImage

model = QwenImage.from_pretrained()                       # same as device_id=0; close() when done

# several prompts: one image each, in order (each result is the same as a single call)
outs = model(["a red bicycle by a stone wall", "a lighthouse at dusk"], seed=[1, 2])
arr = outs[0].numpy()                                     # uint8 [1024, 1024, 3]
rgba = outs[0].numpy(rgba=True)                           # uint8 [1024, 1024, 4] (the model generates alpha)

# diffusers-style names also work
img = model("a lighthouse at dusk", num_inference_steps=40).images[0]

model.close()

Image editing (condition images, editing=True, images=...) is not supported in this package: the code path exists, but it is not tested on this tt-metal build, and with the editing encoders loaded the chip hangs under ETH dispatch.

Use an existing device (the object then does not close it):

import ttnn
from qwen_image_p150 import QwenImage

dev = ttnn.open_mesh_device(ttnn.MeshShape(1, 1), l1_small_size=32768, trace_region_size=640 << 20,
                            dispatch_core_config=ttnn.DispatchCoreConfig(ttnn.DispatchCoreType.ETH))
model = QwenImage.from_pretrained(device=dev)
...
model.close()
ttnn.close_mesh_device(dev)

API reference

QwenImage.from_pretrained(weights=None, *, device_id=0, device=None, dispatch="auto", editing=False, size=1024, revision=None, dit_dtype="bf16", te_dtype="bfp8", warmup=True, warmup_variants=None, verbose=False, download=True, trace_region_size=None)

Opens the device, loads the text encoder (Qwen3-VL-8B language model), the 7B DiT and the VAE decoder onto the chip, compiles every program and captures the metal traces. Returns a QwenImage.

argument meaning
weights None: Qwen/Qwen-Image-2.1 at the validated revision, from the HF cache (downloaded when missing; QWEN_IMAGE_SNAPSHOT=<dir> overrides). Can also be a local snapshot directory or another HF repo id with the same layout.
device_id Chip index among the visible devices (TT_VISIBLE_DEVICES).
device An open 1x1 ttnn mesh device to use. It needs trace_region_size >= 640 MiB and l1_small_size=32768. close() does not close it.
dispatch "auto": ETH dispatch, so all 120 Tensix cores compute, with the worker grid capped at 12x10, as served. Falls back to Tensix dispatch with a RuntimeWarning if this tt-metal cannot open ETH dispatch. Uses Tensix dispatch when editing=True. Other values: "eth", "tensix" (explicit opt-in: 11x10 on a p150). Always 1 command queue. QWEN_IMAGE_ETH_DISPATCH=0/1 overrides "auto".
editing Also loads the vision tower and the VAE encoder so that calls accept images. Not supported in this package (see above).
size Output side for text-to-image. Only 1024 is validated and has tuned matmul blockings.
dit_dtype, te_dtype Weight formats: "bf16" / "bfp8". The defaults (bf16 DiT, bfp8 text encoder) are the published precision policy.
warmup Prepares the request variants of warmup_variants during the load, so that the first real call is as fast as later calls. False: no warm-up (the first call is then slower). See Warm-up.
warmup_variants List of variant dicts for model.warmup(**variant). None (default): see Warm-up. [] is the same as warmup=False.
verbose False (default): a quiet console. tt-metal info and warning lines and ttnn debug lines are hidden (TT_LOGGER_LEVEL=error, LOGURU_LEVEL=WARNING, set only when you did not set them, and only before ttnn is first imported). Errors and Python warnings still show. True: all log lines.
download Set to False to fail instead of downloading missing weights.

model(prompt, *, seed=42, num_steps=40, images=None, num_inference_steps=None, return_latents=False)

argument meaning
prompt str. A list of str returns a list (see generate_many). Prompts up to 256 tokens use the pre-compiled path. The 4 most recent prompts stay on the device, so a repeated prompt skips text encoding and the prefix pass. Longer prompts work, but use a slower path.
seed int: seed of the initial noise (CPU torch.randn in bf16, the same noise as diffusers). For a list of prompts, one int for all prompts or a list with one seed per prompt.
num_steps Euler steps, 1..100 (40 is validated). num_inference_steps is an alias.
images Condition images for editing (not supported in this package; one image or a list, max 4). Each image can be a PIL.Image, a file path, encoded PNG/JPEG bytes, a numpy.ndarray, or a torch.Tensor. Arrays and tensors: HxW, HxWx3, HxWx4 or CxHxW, uint8 or float in [0, 1]. All images are converted to RGBA, as the server does. Each image is resized to about 1 MP. The output has the aspect ratio of the last image.
return_latents Also returns the final latents (one more device read).

Returns QwenImageOutput:

field type
image PIL.Image RGB, composited over white. This is the server's image. 1024x1024 for text-to-image.
image_rgba PIL.Image RGBA, the 4-channel VAE output.
images [image], the diffusers convention.
numpy(rgba=False) uint8 array [H, W, 3] or [H, W, 4].
save(path, rgba=False) Writes the image to a file.
size (width, height).
prompt, seed, num_steps The request.
timings Seconds: encode_s, prefix_s (0 when the prompt is resident), denoise_s (40 steps, with the VAE decode in the same trace), decode_s (image download + PIL), total_s, call_s.
latents torch.Tensor [1, H/16*W/16, 64] bf16 (only with return_latents=True).

model.warmup(num_steps=40, *, return_latents=False, images=None)

Prepares one request variant, so that its first real call is as fast as later calls. A variant that is already prepared costs nothing (idempotent). Outputs do not change. Returns a dict of the work done and its time in seconds.

argument meaning
num_steps A step count or a list. The first text-to-image warm-up runs one full generation of a dummy prompt at the first step count. Each listed step count also gets its host-side schedule prepared.
return_latents Also runs the latents readback once.
images Editing only (not supported in this package): the number (1..4) of 1:1 condition images for one dummy edit.

model.generate_many(prompts, *, seed=42, num_steps=40, images=None, return_latents=False)

Generates one image per prompt, in order, with batch 1 on the device. Each result is identical to a single call with the same prompt and seed. Returns a list of QwenImageOutput.

Lifetime and other members

  • model.close() releases all metal traces and closes the device if from_pretrained opened it. You can call it more than once. with QwenImage.from_pretrained() as model: calls it at the end of the block. At interpreter exit, a model that is still open is closed.
  • One object owns one chip. Calls are serialised with a lock, so you can share one object between threads.
  • model.info: weights path, worker grid, dispatch, load time, warm-up time and the prepared variants (warmed).
  • model.pipeline and model.device: the port's QwenImage21Pipeline and the ttnn device, for advanced use.
  • qwen_image_p150.load_image(x): the image conversion that the API uses (gives PIL RGBA).

Warm-up

from_pretrained prepares the request variants that most users need, so that the first real call is as fast as the later calls. The warm-up does not change outputs. It does not keep results: the warm-up prompt is removed from the prompt cache after the warm-up, so no real call can get a cached result from it.

What changes the work of a call

request parameter what changes on the device / host prepared by the default warm-up?
prompt length 1..256 tokens text-encoder and attention shapes per 32-token bucket yes: compiled for all lengths at load time (QWEN_PREWARM=1), with or without warm-up
new prompt text text encoder, prefix pass, one step-trace capture every new prompt does this work; it is part of the steady-state time, not a first-call cost
first request of the process tokenizer load, step-trace and VAE-trace capture, host buffers yes: one 40-step generation of a dummy prompt
num_steps the host-side schedule (modulation rows) is computed once per step count and uploaded yes for 40, 50, 30 and 20; other counts compute it on their first call
return_latents=True one more device read yes (the dummy generation reads the latents)
seed, list of prompts nothing new not needed
prompt longer than 256 tokens variable-shape path: new programs for each new length, release of all traces no (see below)

Defaults and options

model = QwenImage.from_pretrained()                                   # default warm-up
model = QwenImage.from_pretrained(warmup_variants=[{"num_steps": [40, 25, 60]}])   # other step counts
model = QwenImage.from_pretrained(warmup=False)                       # fastest load, slow first call
model.warmup(num_steps=8)                                             # prepare one more variant later
model.warmup(num_steps=8)                                             # again: no work (idempotent)
print(model.info["warmed"], model.info["warmup_s"])

The default is warmup_variants=[{"num_steps": [40, 50, 30, 20], "return_latents": True}], plus {"images": 1} first when editing=True. The default text-to-image warm-up is one 40-step generation, the first trace captures and the four host schedules. Host memory: about 80 KB per prepared step, 11 MB for the default. Device memory: no extra DRAM and no extra trace (the warm-up uses the same step trace and VAE trace as any call; its prompt slot is freed afterwards).

Variants that cannot be prepared

  • Prompts longer than 256 tokens. They use a variable-shape path: each new length compiles new programs (text encoder per 32-token bucket, prefix and attention per exact length), releases all traces and captures a 40-step loop trace, so every such call is slower than a call with a short prompt, on the first call and on later calls; their speed is not measured. The length is not bounded (2000 characters can be more than 2000 tokens), so it cannot be compiled in advance. Keep prompts at 256 tokens or fewer.
  • Step counts that are not prepared compute their host-side schedule on their first call. Use warmup_variants or model.warmup(num_steps=...) for the step counts that you use.

Notes

  • Only batch 1 and no classifier-free guidance. This is how the model is sampled.
  • The package contains the port's models/ tree (models.experimental.qwen_image_2_1 and modified copies of some models.tt_dit / models.common files from tt-metal). When you import qwen_image_p150, these copies are put first in the models namespace, also when $TT_METAL_HOME is on PYTHONPATH. Import qwen_image_p150 before other tt-metal models.* modules in the same process.
  • The precision and performance settings (QWEN_* environment variables, listed under container.serve.env in tt_kernel_manifest.json) keep their served defaults (the container sets them explicitly). Do not change them unless you want to experiment.
  • Tests: TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py (no device; needs pytest) and python -m pytest code/tests/test_api_device.py -s (one chip, weights in the HF cache). The device test starts with test_first_call_fast: after the default warm-up, the first real call (40 steps, and 20 steps) must be within 10 % of the later calls. code/tests/bench_first_call.py measures first-call latency per variant in a fresh process.