Please support INT8 Model loader for x2 Speed boost
I get a 100% speed boost, x2 with a INT8 LTX 2.3 model. Generation speed halved on a rtx 3060. Can u please support INT8 model loader + lora support? That would be a dream. There are already different INT8 models: https://huggingface.co/bertbobson/ComfyUI-INT8_ConvRot/tree/main and iam using the custom node from here https://github.com/overpresentme/ComfyUI-ltx-int8-loader/ to load the model, u just replace the model loader. Only problem: Loras do not work. There is another one here which work with loras https://github.com/BobJohnson24/ComfyUI-INT8-Fast BUT: Model initialization is very very slow, longer than the generation itself, idk why. I hope u can do it, thank you from my heart.
Got an old rusty 3xxx myself, so i tried that INT8 loader actually... For sure a speed-up at the actual inference...
I didnt try any loras though.
Something more robust and native to comfy would be great, but from the fraction of comments here and there, i think maybe its not as easy as it might sound. And FP8 being the better alternative.
But Kijai will know better ;-)
For GPUs that doesn't support FP8 natively, INT8 will certainly be faster 🤔 since FP8 will be upcasted to BF16/FP16.
+1 for INT8! Having mixed results with the other loader/models posted out there. The speed boost is significant for the 3xxx cards.
+1
If you don't have many LORAs, or if you frequently use the same ones, you can bake them into the model at the desired precision and then quantize them to INT8.
This is, of course, not for testing, but for actual production use.
The standard workflow will run smoothly, quickly, and efficiently
Here's the time it takes for all these conversions and for creating the image.
[INFO] got prompt
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type FLUX
[INFO] Requested to load Flux2
[INFO] Model Flux2 prepared for dynamic VRAM loading. 61461MB Staged. 138 patches attached. Force pre-loaded 128 weights: 71 KB.
[INFO] Prompt executed in 146.57 seconds
[INFO] got prompt
[INFO] INT8 Fast: Aimdo dynamic loading active, using cuda:0 as a per-layer bake/quant work device.
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type FLUX
INT8 Fast: Quantizing on-the-fly (ConvRot: True)
[INFO] Requested to load Flux2
[INFO] Model Flux2 prepared for dynamic VRAM loading. 31436MB Staged. 0 patches attached. Force pre-loaded 128 weights: 15863 KB.
[INFO] Model Flux2 prepared for dynamic VRAM loading. 31436MB Staged. 0 patches attached. Force pre-loaded 128 weights: 15863 KB.
[INFO] Prompt executed in 233.34 seconds
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] Found quantization metadata version 1
[INFO] Using MixedPrecisionOps for text encoder
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load Flux2TEModel_
[INFO] Model Flux2TEModel_ prepared for dynamic VRAM loading. 17180MB Staged. 0 patches attached. Force pre-loaded 270 weights: 600 KB.
[INFO] Found quantization metadata version 1
[INFO] Detected mixed precision quantization
[INFO] model weight dtype torch.float16, manual cast: torch.float16
[INFO] model_type FLUX
[INFO] Requested to load Flux2
[INFO] 0 models unloaded.
[INFO] Model Flux2 prepared for dynamic VRAM loading. 31436MB Staged. 0 patches attached. Force pre-loaded 128 weights: 15863 KB.
100%|████████| 8/8 [00:24<00:00, 3.05s/it]
[INFO] Requested to load AutoencoderKL
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached. Force pre-loaded 110 weights: 187 KB.
[INFO] Prompt executed in 73.65 seconds
This isn't really a solution, of course, but it's an option if you really need Loras.
native INT8 support in ComfyUI now ;-) (can use regular normal model loader node)
And model already available here https://huggingface.co/Kijai/LTX2.3_comfy/tree/main/diffusion_models
You are the best. It is working like a charm. Can we have wan2.2 i2v/s2v int8, please ?
thanks to Kijai and the Comfy team ;-) not me ..
(just posted to make you aware of the new thing.. ;-))
