akhaliq/Qwen-Image-2.1-Next-Scene-LoRA

🎥 Qwen-Image-2.1 · Next-Scene LoRA

Feed a frame, direct the next shot. This LoRA teaches Qwen-Image-2.1 cinematic scene continuation: it takes an input image and a director-style instruction and generates the next scene — camera moves, reveals, lighting shifts — while keeping the world, characters and style consistent. Chain outputs into storyboards.

This is the Qwen-Image-2.1 port of the next-scene format pioneered by lovis93/next-scene-qwen-image-lora-2509 (313K downloads on the 2509 edit model) — retrained natively on the newest Qwen image base, where this capability did not exist yet. Companion piece to akhaliq/Qwen-Image-2.1-Multiple-Angles-LoRA (camera control) — together they cover where the camera is and where the story goes next.

Usage

from diffusers import QwenImage21Pipeline
import torch

pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", torch_dtype=torch.bfloat16)
pipe.load_lora_weights(
    "akhaliq/Qwen-Image-2.1-Next-Scene-LoRA",
    weight_name="next_scene_step2500.safetensors",  # diffusers/ComfyUI-converted
)

image = pipe(
    prompt="Next Scene: The camera pulls back to a sweeping aerial view, "
           "revealing the fleet behind the cliffs. Soft morning light, atmospheric depth.",
    image=input_image,          # your current frame
    num_inference_steps=28,
    true_cfg_scale=1.0,
).images[0]

ComfyUI: load in any Qwen-Image-2.1 workflow with a LoRA node (strength 0.7–0.9), pass your image as the edit/reference input.

Prompt format: start with Next Scene: and lead with the camera direction:

Next Scene: The camera tracks forward and tilts down as rain begins to streak the lens.
Next Scene: Cut to a low-angle shot; sunlight breaks through the clouds behind her.
Next Scene: The camera pans right, revealing the fleet massing behind the ridge.

Pro tips

• Begin with the camera move, then the subject/lighting change — the model was trained on captions written this way.
• Chain generations (output → input) to build sequential storyboards.
• If the output drifts toward copying the input, lower the LoRA strength toward 0.7.

🎬 Next Scene chain — one photo, four directed shots

Starting from a single photograph, each frame is the previous frame's output fed back in with a new director's instruction. Step-2,500 checkpoint, strength 0.7, 40 steps per hop, a grain-breaking blur on chained inputs (see the chaining notes in Scope & limits):

Starting frame: Aldrin Apollo 11 (NASA, public domain). The hops:

1. "…pulls back to a wide shot, revealing the vast lunar plain…"
2. "…pans right as the astronaut bounds across the regolith, Earth rising over the horizon."
3. "Cut to a low-angle shot as the astronaut salutes the flag…"
4. "…pushes in on the astronaut's gold visor, the lunar module reflected in the curved glass."

Same world, same astronaut, four directed shots — no video model, just chained edits. Individual frames: chain/hop_*.jpg.

🎬 Checkpoint sweep (cinematic storm test)

One cinematic starter (generated by the base model), then the same dramatic instruction across checkpoints — "Next Scene: The camera pulls back to an extreme wide shot as a violent storm rolls over the valley, lightning splitting the sky above the ruins, rain sweeping across the frame, the explorer silhouetted against the flash." — 40 steps, strength 0.9, seed 42, plus a no-LoRA control:

Read it row by row: the no-LoRA base already attempts the storm (its editing is strong), but the trained checkpoints push the transformation much harder — step 2500 rolls a full storm front over the valley while keeping the explorer, ruins and palette intact. Individual cells + starter: sweep/.

Which checkpoint

Checkpoint	Verdict
next_scene_step2500.safetensors (repo root)	Recommended. Final step, converted to diffusers key layout. Strongest scene transformation in the sweep (see grid above) while holding composition and identity.
next_scene_step1500.safetensors	Softer alternative — moderate transformation, gentler on identity. Try this if 2500 pushes too hard.
next_scene_step1000.safetensors	Weakest transformation — mostly for studying the training curve.
checkpoints/qwen21_next_scene_v1_*.safetensors	Raw ai-toolkit format — the MLP keys use ai-toolkit's fused naming, which diffusers silently skips. Only use with ai-toolkit/its own loader. (qwen21_next_scene_v1.safetensors = step 2500.)

Per-250-step sample grids from the training run are in checkpoints/samples/ so the training curve is inspectable end to end.

Training

• Base: Qwen/Qwen-Image-2.1 (unified generate + edit), arch: qwen_image_2
• Trainer: ostris/ai-toolkit — rank 64 / alpha 64, adamw8bit, lr 1e-4, weighted timesteps, caption dropout 0.05, 512², 2,500 steps, uint3-quantized base, bf16 training
• Data: 1,144 scene-progression pairs — consecutive frames from the Blender open films Sintel and Tears of Steel (© Blender Foundation, CC-BY, via KevinConnorLee/Sintel (MPI-Sintel frames) and the Blender mirrors). Captions are director-style "Next Scene: …" instructions written by Qwen3-VL-4B over each (frame → next frame) pair. Published: pairs · captioned training set
• Dashboard: trackio training curves
• Demo: akhaliq/qwen21-next-scene

Scope & limits

• Trained on cinematic film frames (one animated, one live-action film) — it generalizes to photographic inputs, but the strongest results are on filmic scenes, landscapes and establishing shots; static portraits are out of scope by design.
• Trained at 512²; larger canvases inherit the behavior but were not trained on.
• Chaining notes (measured, not guessed): naive chaining at strength 0.9 accumulates film grain into visible color noise by hop 3–4. The verified recipe for clean chains is strength ≤ 0.7 + a 0.5px gaussian blur on each chained input (breaks the grain-amplification feedback loop) — the storyboard above is rendered exactly this way, and the demo Space applies it automatically. Baseline vs mitigation grids: chain/ vs mitigations/. Late-hop transformations stay conservative — that is a data limitation of v1, not a settings issue.
• This is a v1: 1.1K pairs from two films. A v2 with broader footage is the planned upgrade.
• Checkpoint selection here follows the mid-training-peak heuristic from prior Qwen-Image-2.1 LoRA runs on this trainer; judge checkpoints/samples/ grids and pick your own if you prefer a stronger or lighter bake.