gguf-org/pig-clip

🐷pig architecture gguf clip

• natively trained multipurpose clip for gk engine
• low vram alternative/substitute of t5xxl, umt5xxl, etc.
• 80-90% smaller, better efficient and cost effective (hopefully)

rationale

since LLM in diffusion model basically serves as text encoder, which means its primary goal/function is delivering a prompt to the generator precisely, no talk, no deep reasoning, no long response, make it as simple as possible

limitation

we stop the training when similarity reaches .9 or over 20k cycles and similarity not less than .8

how it works

simply replace --t5xxl t5xxl.gguf with --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf

examples

test it with pixart

ggk diffuser engine -- --diffusion-model pixart-nvfp4.gguf --vae pig_pixart_vae_fp16-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --diffusion-fa -v -o out.png

test it with sd-lite

ggk diffuser engine -- --diffusion-model model.gguf --vae vae.gguf --clip_l clip_l.gguf --clip_g clip_g.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --steps 8 --cfg-scale 1 --sampling-method euler --clip-on-cpu --diffusion-fa -v -o out.png

umt5 adapter

test it with wan2.1 or wan2.2

ggk diffuser engine -- -M vid_gen --diffusion-model wan2.1_t2v_1.3b-q4_0.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_umt5_adapter-f16.gguf -p "a pig moving quickly in a beautiful winter scenery nature trees sunset tracking camera" --cfg-scale 6.0 --sampling-method euler -v -n "blurry ugly bad" -W 480 -H 480 --diffusion-fa --offload-to-cpu --video-frames 14 -o out.avi

qwen3-4b adapter

test it with x-image

ggk diffuser engine -- --diffusion-model  x_image-nvfp4.gguf --vae pig_flux_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3_4b_adapter-f16.gguf -p "cute anime style girl with pinky messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron, it is a postcard held by a hand in front of a beautiful realistic city at sunset and there is cursive writing that says GGUF" --cfg-scale 1.0 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3vl-4b adapter

test it with mageflow

ggk diffuser engine -- --diffusion-model mageflow-edit-turbo-nvfp4.gguf --vae pig_mageflow_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_4b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-4b-it-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

Reference

pig engine - the new gguf compute kernels (gk)