vit_qat.onnx fails to load in ONNX Runtime β float32/bfloat16 type mismatch cascading through LayerNorm + quantizer nodes
Hi team,
Thanks for publishing GR00T-N1.6-bridge-INT8-Edge β really useful reference for INT8 quantization work on GR00T.
I've been trying to build TensorRT engines from the onnx/ artifacts (vit_qat.onnx, llm_qat.onnx, dit_qat.onnx) to reproduce your benchmark numbers, and have hit a specific issue I'd appreciate your input on:
Loading vit_qat.onnx directly in ONNX Runtime (CPU provider) fails with a type mismatch error at the first LayerNormalization node:
"Type parameter (T) of Optype (LayerNormalization) bound to different types (tensor(float) and tensor(bfloat16))"
When I patch that node with an inserted Cast, the same mismatch reappears one node downstream at the next input_quantizer Mul op β suggesting activations are float32 while weights/scales are bfloat16 throughout much of the QAT-simulation subgraph, without bridging casts in between.
TensorRT itself builds an engine from the unpatched graph without complaint, but I haven't been able to verify numerical correctness against the PyTorch reference β outputs diverge significantly (cosine similarity ~0.5), which could be downstream of this same dtype issue.
A few questions:
- Is this a known limitation of how these ONNX files were exported (e.g., an artifact of the QAT training precision setup), or does it indicate something broken in the exported files?
- Do you have the actual inference/build script you used to produce your reported benchmark numbers (12.3 FPS / 81.3ms on Jetson AGX Orin)? Having the reference pipeline would help enormously in confirming we're consuming these ONNX exports correctly.
- Would you be able to share whether pixel_values for the ViT export expects any preprocessing beyond a plain [B, 3, 252, 252] tensor (e.g., normalization stats, or the NaFlex-style [B, N, C, H, W] format used elsewhere in Siglip2)?
Happy to share our test scripts and full logs if useful for reproducing on your end. Thanks for any pointers!
Hi, thanks for the detailed report, and sorry for the slow answer.
This repo is an older experiment from our team: ModelOpt PTQ + QAT with fake-quant ONNX. We've stopped maintaining it, so here are short answers, then where we'd point you instead.
The dtype mismatch comes from how these files were exported, not from file corruption. The model was cast to bf16 and traced with ModelOpt's simulated quantizers (opset 19). Some intermediates come out as float32 next to bf16 constants. TensorRT accepts that graph, but ONNX Runtime's strict type checking rejects it, and we never validated these files in ORT. Patching casts node by node will keep running into the same issue.
The build script. The engines behind the 12.3 FPS / 81.3 ms numbers were built with an internal script that we never published and won't maintain. Since your TRT outputs reach only about 0.5 cosine against PyTorch, I'd rather you didn't spend more time reproducing this pipeline.
pixel_values is a single bf16 tensor [B, 3, 252, 252], the output of the checkpoint's own image processor (resize + normalize). It is not the NaFlex [B, N, C, H, W] layout.
Our current work is FoldQuantVLA, which replaces this approach: https://github.com/cair-vinuni/FoldQuantVLA Β· paper: https://arxiv.org/abs/2609.24433
- Projections run as real INT8 / INT4 on the tensor cores through custom TensorRT plugins, not as Q/DQ simulation. No QAT or retraining is needed, just about 128 calibration frames.
- The whole path is public and documented: quantize β export β build_engines β verify β serve. verify compares the engines against the PyTorch policy, so a ~0.5 cosine like yours shows up before deployment.
- It covers GR00T N1.5 / N1.6 / N1.7 and Οβ.β , on Jetson AGX Orin and on Ada desktop GPUs.
- On GR00T N1.6 LIBERO (800 episodes), W4A4 with the LLM's o_proj/down_proj kept at INT8 scores 96.62% vs 96.38% bf16. W4A4 is 1.20β1.33Γ faster than float TensorRT on Orin.
Quantized checkpoints, ready to serve or to build engines from, are in our collection: https://hf.co/collections/vrfai/foldquantvla. For N1.6, start with vrfai/GR00T-N1.6-LIBERO-w4a4-res8.
If you're on the Bridge / WidowX setup, FoldQuantVLA's pipeline works on your own fine-tuned N1.6 checkpoint too. Open an issue on the GitHub repo if anything breaks; we answer there faster.