Download code/PYTHON.md from changh95/qwen-image-2.1-p150: direct link, hf CLI and curl.
- Browser
- Download file 14.9 kB
-
https://huggingface.co/changh95/qwen-image-2.1-p150/resolve/main/code/PYTHON.md
- Command line
-
hf download hf://changh95/qwen-image-2.1-p150/code/PYTHON.md
-
curl -L -o PYTHON.md https://huggingface.co/changh95/qwen-image-2.1-p150/resolve/main/code/PYTHON.md
qwen-image-p150: Python API
Use Qwen-Image-2.1 text-to-image on one Tenstorrent Blackhole p150 from Python code: scripts, pipelines and
notebooks. The package wraps the port in models/experimental/qwen_image_2_1. It is the same pipeline that the
HTTP server of the container uses, with the same defaults, so a call gives the same pixels as POST /predict.
Install
You need an environment that already has tt-metal (ttnn), built from tt-metal commit 8b98410e730b with the
patch in patches/ of this repository applied (it adds dispatch on the chip's idle Ethernet cores). The default
device configuration is ETH dispatch, 1 command queue and a 12x10 worker grid; the model uses only command
queue 0. Without ETH dispatch support the package falls back to Tensix dispatch and gives a RuntimeWarning: a
p150 then has an 11x10 worker grid and each step is slower.
git clone https://huggingface.co/changh95/qwen-image-2.1-p150
export PYTHONPATH=$PWD/qwen-image-2.1-p150/code:$PYTHONPATH # the package and its bundled models/ tree
pip install "numpy<2" safetensors "transformers==5.12.1" huggingface_hub loguru pillow # host-side dependencies
code/ is used from the checkout (it is not a pip package). It does not install or change ttnn. The HTTP server
(models/experimental/qwen_image_2_1/server/app.py) also needs fastapi, uvicorn and pydantic; the tests need
pytest.
cd qwen-image-2.1-p150
TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py -q # host tests only: no device
python -m pytest code/tests/test_api_device.py -s # device tests: one chip
TT_VISIBLE_DEVICES=none makes sure that the host tests cannot open a device. They do not need the weights.
Quickstart
from qwen_image_p150 import QwenImage
with QwenImage.from_pretrained(device_id=0) as model:
out = model("White furry llama with black sunglasses, smiling and happy, jumping", seed=42, num_steps=40)
out.image.save("llama.png") # PIL.Image, 1024x1024 RGB
print(out.timings) # seconds per stage
The first from_pretrained on a machine downloads the weights (Qwen/Qwen-Image-2.1 at revision 790c926,
about 33 GB) into the Hugging Face cache and compiles the kernels, which takes many minutes. Later loads read the
weights, pre-compile the programs for all prompt lengths and run the warm-up (see Warm-up), which makes
the first call as fast as the later calls. Speed: see "Expected performance" on the model card;
code/scripts/bench_inproc.py measures it.
The console stays quiet by default (verbose=False): tt-metal info and warning lines and the load progress lines
are not shown. Errors and Python warnings still show. Use from_pretrained(verbose=True) to see all log lines.
code/examples/quickstart.py runs this snippet and writes llama.png and llama.json (the timings).
More examples
import numpy as np
from qwen_image_p150 import QwenImage
model = QwenImage.from_pretrained() # same as device_id=0; close() when done
# several prompts: one image each, in order (each result is the same as a single call)
outs = model(["a red bicycle by a stone wall", "a lighthouse at dusk"], seed=[1, 2])
arr = outs[0].numpy() # uint8 [1024, 1024, 3]
rgba = outs[0].numpy(rgba=True) # uint8 [1024, 1024, 4] (the model generates alpha)
# diffusers-style names also work
img = model("a lighthouse at dusk", num_inference_steps=40).images[0]
model.close()
Image editing (condition images, editing=True, images=...) is not supported in this package: the code path
exists, but it is not tested on this tt-metal build, and with the editing encoders loaded the chip hangs under ETH
dispatch.
Use an existing device (the object then does not close it):
import ttnn
from qwen_image_p150 import QwenImage
dev = ttnn.open_mesh_device(ttnn.MeshShape(1, 1), l1_small_size=32768, trace_region_size=640 << 20,
dispatch_core_config=ttnn.DispatchCoreConfig(ttnn.DispatchCoreType.ETH))
model = QwenImage.from_pretrained(device=dev)
...
model.close()
ttnn.close_mesh_device(dev)
API reference
QwenImage.from_pretrained(weights=None, *, device_id=0, device=None, dispatch="auto", editing=False, size=1024, revision=None, dit_dtype="bf16", te_dtype="bfp8", warmup=True, warmup_variants=None, verbose=False, download=True, trace_region_size=None)
Opens the device, loads the text encoder (Qwen3-VL-8B language model), the 7B DiT and the VAE decoder onto the
chip, compiles every program and captures the metal traces. Returns a QwenImage.
| argument | meaning |
|---|---|
weights |
None: Qwen/Qwen-Image-2.1 at the validated revision, from the HF cache (downloaded when missing; QWEN_IMAGE_SNAPSHOT=<dir> overrides). Can also be a local snapshot directory or another HF repo id with the same layout. |
device_id |
Chip index among the visible devices (TT_VISIBLE_DEVICES). |
device |
An open 1x1 ttnn mesh device to use. It needs trace_region_size >= 640 MiB and l1_small_size=32768. close() does not close it. |
dispatch |
"auto": ETH dispatch, so all 120 Tensix cores compute, with the worker grid capped at 12x10, as served. Falls back to Tensix dispatch with a RuntimeWarning if this tt-metal cannot open ETH dispatch. Uses Tensix dispatch when editing=True. Other values: "eth", "tensix" (explicit opt-in: 11x10 on a p150). Always 1 command queue. QWEN_IMAGE_ETH_DISPATCH=0/1 overrides "auto". |
editing |
Also loads the vision tower and the VAE encoder so that calls accept images. Not supported in this package (see above). |
size |
Output side for text-to-image. Only 1024 is validated and has tuned matmul blockings. |
dit_dtype, te_dtype |
Weight formats: "bf16" / "bfp8". The defaults (bf16 DiT, bfp8 text encoder) are the published precision policy. |
warmup |
Prepares the request variants of warmup_variants during the load, so that the first real call is as fast as later calls. False: no warm-up (the first call is then slower). See Warm-up. |
warmup_variants |
List of variant dicts for model.warmup(**variant). None (default): see Warm-up. [] is the same as warmup=False. |
verbose |
False (default): a quiet console. tt-metal info and warning lines and ttnn debug lines are hidden (TT_LOGGER_LEVEL=error, LOGURU_LEVEL=WARNING, set only when you did not set them, and only before ttnn is first imported). Errors and Python warnings still show. True: all log lines. |
download |
Set to False to fail instead of downloading missing weights. |
model(prompt, *, seed=42, num_steps=40, images=None, num_inference_steps=None, return_latents=False)
| argument | meaning |
|---|---|
prompt |
str. A list of str returns a list (see generate_many). Prompts up to 256 tokens use the pre-compiled path. The 4 most recent prompts stay on the device, so a repeated prompt skips text encoding and the prefix pass. Longer prompts work, but use a slower path. |
seed |
int: seed of the initial noise (CPU torch.randn in bf16, the same noise as diffusers). For a list of prompts, one int for all prompts or a list with one seed per prompt. |
num_steps |
Euler steps, 1..100 (40 is validated). num_inference_steps is an alias. |
images |
Condition images for editing (not supported in this package; one image or a list, max 4). Each image can be a PIL.Image, a file path, encoded PNG/JPEG bytes, a numpy.ndarray, or a torch.Tensor. Arrays and tensors: HxW, HxWx3, HxWx4 or CxHxW, uint8 or float in [0, 1]. All images are converted to RGBA, as the server does. Each image is resized to about 1 MP. The output has the aspect ratio of the last image. |
return_latents |
Also returns the final latents (one more device read). |
Returns QwenImageOutput:
| field | type |
|---|---|
image |
PIL.Image RGB, composited over white. This is the server's image. 1024x1024 for text-to-image. |
image_rgba |
PIL.Image RGBA, the 4-channel VAE output. |
images |
[image], the diffusers convention. |
numpy(rgba=False) |
uint8 array [H, W, 3] or [H, W, 4]. |
save(path, rgba=False) |
Writes the image to a file. |
size |
(width, height). |
prompt, seed, num_steps |
The request. |
timings |
Seconds: encode_s, prefix_s (0 when the prompt is resident), denoise_s (40 steps, with the VAE decode in the same trace), decode_s (image download + PIL), total_s, call_s. |
latents |
torch.Tensor [1, H/16*W/16, 64] bf16 (only with return_latents=True). |
model.warmup(num_steps=40, *, return_latents=False, images=None)
Prepares one request variant, so that its first real call is as fast as later calls. A variant that is already prepared costs nothing (idempotent). Outputs do not change. Returns a dict of the work done and its time in seconds.
| argument | meaning |
|---|---|
num_steps |
A step count or a list. The first text-to-image warm-up runs one full generation of a dummy prompt at the first step count. Each listed step count also gets its host-side schedule prepared. |
return_latents |
Also runs the latents readback once. |
images |
Editing only (not supported in this package): the number (1..4) of 1:1 condition images for one dummy edit. |
model.generate_many(prompts, *, seed=42, num_steps=40, images=None, return_latents=False)
Generates one image per prompt, in order, with batch 1 on the device. Each result is identical to a single call
with the same prompt and seed. Returns a list of QwenImageOutput.
Lifetime and other members
model.close()releases all metal traces and closes the device iffrom_pretrainedopened it. You can call it more than once.with QwenImage.from_pretrained() as model:calls it at the end of the block. At interpreter exit, a model that is still open is closed.- One object owns one chip. Calls are serialised with a lock, so you can share one object between threads.
model.info: weights path, worker grid, dispatch, load time, warm-up time and the prepared variants (warmed).model.pipelineandmodel.device: the port'sQwenImage21Pipelineand the ttnn device, for advanced use.qwen_image_p150.load_image(x): the image conversion that the API uses (gives PIL RGBA).
Warm-up
from_pretrained prepares the request variants that most users need, so that the first real call is as fast as
the later calls. The warm-up does not change outputs. It does not keep results: the warm-up
prompt is removed from the prompt cache after the warm-up, so no real call can get a cached result from it.
What changes the work of a call
| request parameter | what changes on the device / host | prepared by the default warm-up? |
|---|---|---|
| prompt length 1..256 tokens | text-encoder and attention shapes per 32-token bucket | yes: compiled for all lengths at load time (QWEN_PREWARM=1), with or without warm-up |
| new prompt text | text encoder, prefix pass, one step-trace capture | every new prompt does this work; it is part of the steady-state time, not a first-call cost |
| first request of the process | tokenizer load, step-trace and VAE-trace capture, host buffers | yes: one 40-step generation of a dummy prompt |
num_steps |
the host-side schedule (modulation rows) is computed once per step count and uploaded | yes for 40, 50, 30 and 20; other counts compute it on their first call |
return_latents=True |
one more device read | yes (the dummy generation reads the latents) |
seed, list of prompts |
nothing new | not needed |
| prompt longer than 256 tokens | variable-shape path: new programs for each new length, release of all traces | no (see below) |
Defaults and options
model = QwenImage.from_pretrained() # default warm-up
model = QwenImage.from_pretrained(warmup_variants=[{"num_steps": [40, 25, 60]}]) # other step counts
model = QwenImage.from_pretrained(warmup=False) # fastest load, slow first call
model.warmup(num_steps=8) # prepare one more variant later
model.warmup(num_steps=8) # again: no work (idempotent)
print(model.info["warmed"], model.info["warmup_s"])
The default is warmup_variants=[{"num_steps": [40, 50, 30, 20], "return_latents": True}], plus
{"images": 1} first when editing=True. The default text-to-image warm-up is one 40-step generation, the
first trace captures and the four host schedules. Host memory: about 80 KB per prepared step, 11 MB for the
default. Device memory: no extra DRAM and no extra trace (the warm-up uses the same step trace and VAE trace as
any call; its prompt slot is freed afterwards).
Variants that cannot be prepared
- Prompts longer than 256 tokens. They use a variable-shape path: each new length compiles new programs (text encoder per 32-token bucket, prefix and attention per exact length), releases all traces and captures a 40-step loop trace, so every such call is slower than a call with a short prompt, on the first call and on later calls; their speed is not measured. The length is not bounded (2000 characters can be more than 2000 tokens), so it cannot be compiled in advance. Keep prompts at 256 tokens or fewer.
- Step counts that are not prepared compute their host-side schedule on their first call. Use
warmup_variantsormodel.warmup(num_steps=...)for the step counts that you use.
Notes
- Only batch 1 and no classifier-free guidance. This is how the model is sampled.
- The package contains the port's
models/tree (models.experimental.qwen_image_2_1and modified copies of somemodels.tt_dit/models.commonfiles from tt-metal). When you importqwen_image_p150, these copies are put first in themodelsnamespace, also when$TT_METAL_HOMEis onPYTHONPATH. Importqwen_image_p150before other tt-metalmodels.*modules in the same process. - The precision and performance settings (
QWEN_*environment variables, listed undercontainer.serve.envintt_kernel_manifest.json) keep their served defaults (the container sets them explicitly). Do not change them unless you want to experiment. - Tests:
TT_VISIBLE_DEVICES=none python -m pytest code/tests/test_api_host.py(no device; needs pytest) andpython -m pytest code/tests/test_api_device.py -s(one chip, weights in the HF cache). The device test starts withtest_first_call_fast: after the default warm-up, the first real call (40 steps, and 20 steps) must be within 10 % of the later calls.code/tests/bench_first_call.pymeasures first-call latency per variant in a fresh process.