marcoyang/salmonn-2-8b-test

SALMONN-2 8B

SALMONN-2 is an open-source audio large language model (ALLM) for speech, general audio, music, and paralinguistic understanding. It combines the SPEAR audio encoder, a multi-layer feature-fusion adapter, and Qwen3-8B.

This checkpoint contains the merged Qwen LoRA weights, SPEAR audio encoder, audio connector, tokenizer assets, and custom Hugging Face model code. Load it with trust_remote_code=True.

For command-line and batch inference, multi-audio and multimodal in-context learning examples, environment details, and fine-tuning instructions, see the SALMONN-2 GitHub repository.

Results

SALMONN-2 achieves strong performance on three audio-language model (ALLM) benchmarks while using 18.2k hours of supervised audio-text training data. The table below shows the 8B model comparison.

Model	Supervised audio-text data (h)	MMAU-Pro	MMAR	MMSU
Qwen2.5-Omni	--	52.2	56.7	61.3
Kimi-Audio	>13M	56.6	60.8	54.7
MiMo-Audio	>1M	53.4	61.7	61.9
AF-3	>55k	51.7	58.5	61.4
MOSS-Audio	>1M	57.5	64.4	66.4
SALMONN-2 8B	18.2k	58.5	64.5	69.5

Minimal Hugging Face inference

The following example loads this repository directly with Hugging Face transformers; cloning or installing the SALMONN-2 GitHub package is not required.

pip install "transformers>=4.57,<5" accelerate torch torchaudio

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "marcoyang/salmonn-2-8b-test"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()
processor.prepare_model(model)

inputs = processor(
    audios=["example.wav"],
    instruction="Please describe the audio.",
)
device = next(model.parameters()).device

with torch.inference_mode():
    output_ids = model.generate(
        **inputs.to(device),
        max_new_tokens=256,
        do_sample=False,
    )

print(processor.decode(output_ids[0]))

The processor loads and resamples audio, computes the SPEAR filterbank input, formats the chat prompt, and returns all model inputs. The example uses bfloat16 and automatic device placement. A CUDA GPU is recommended for practical inference.

Contextual ASR

For text-only contextual words, pass a list of strings:

inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=["howes", "wszelaki"],
)

To provide both the spelling and pronunciation of each contextual word, pair the text with a context audio file:

inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=[
        {"text": "howes", "audio": "howes.wav"},
        {"text": "wszelaki", "audio": "wszelaki.wav"},
    ],
)

The processor places all model-specific audio markers and contextual formatting automatically.

Advanced prompt placement

For custom multi-audio layouts, use formatted_prompt with one <audio> marker per input file. Audio files are matched to the markers from left to right:

inputs = processor(
    audios=["main.wav", "example.wav"],
    formatted_prompt=(
        "<audio>Compare the main recording with this example: "
        "<audio>What do they have in common?"
    ),
)

The processor still loads the audio, computes filterbanks, validates marker alignment, applies the chat template, and returns model-ready tensors.

License

Apache License 2.0. See LICENSE.

Citation

@inproceedings{yang2026spear,
  title     = {SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations},
  author    = {Yang, Xubo and Yang, Yuxuan and Jin, Ziyang and Cui, Zeyu and Wu, Wen and Li, Bo and Zhang, Chao and Woodland, Philip C.},
  booktitle = {Proceedings of the Forty-third International Conference on Machine Learning},
  year      = {2026}
}