Is this model compatible with Transformers.js?

#2
by mrllmman357 - opened

I'm new to working with AI models, so pardon my ignorance. The Transformers.js documentation says that models must be in ONNX format to work with Transformers.js. Since this model is already in ONNX format, does that mean it'll work if I just convert the Python commands in the model description to JS syntax?

ONNX Community org

Onnx Model By design are meant to be device & language agnostic, so yes this will work. Only thing you have to make sure the input name & dimension are same as expected by model .

ONNX Community org

Not quite—ONNX is necessary here, but it isn't sufficient. OmniVoice's inference.py coordinates several models: the embedding encoder, Qwen3 decoder, audio heads, a 32-step unmasking loop, and the Higgs decoder. Translating the Python calls to JS won't recreate that pipeline by itself.

You'd need to port that orchestration to ONNX Runtime Web and confirm its operators and execution providers work there, or add OmniVoice model/processor support to Transformers.js. Their contribution guide describes ONNX export as only the first of three steps: https://github.com/huggingface/transformers.js/blob/main/CONTRIBUTING.md#do-you-want-to-implement-a-new-model

Thank you both for the detailed responses!

ONNX Community org

@codelion
Okay, I get it now. You are correct — since the whole model is divided into sub-parts (embeddings encoder, Qwen3 decoder, audio heads, and the Higgs DAC-style encoder/decoder/quantizer), not all of them are transformer submodels, and Transformers.js doesn't have a registered architecture that knows how to orchestrate this pipeline (the 32-step iterative-unmasking loop especially — that's custom control flow, not something pipeline() can drive out of the box).

For this scenario, onnxruntime-web is the better solution — it can load and run each ONNX sub-graph directly, and I can write the orchestration loop myself in JS instead of waiting on native Transformers.js support.

ONNX Community org
import * as ort from "onnxruntime-web";                                                // Model execution
import { AutoTokenizer } from "@huggingface/transformers";       //  Data Encoding & Decoding
Prince-1 changed discussion status to closed

Sign up or log in to comment