---
license: apache-2.0
library_name: transformers
tags:
  - audio
  - audio-language-model
  - audio-understanding
  - speech
  - music
---

# SALMONN-2 8B

SALMONN-2 is an open-source audio large language model (ALLM) for speech, general audio, music, and
paralinguistic understanding. It combines the
[SPEAR audio encoder](https://huggingface.co/marcoyang/spear-xlarge-speech-audio-v2), a multi-layer
feature-fusion adapter, and Qwen3-8B.

This checkpoint contains the merged Qwen LoRA weights, SPEAR audio encoder, audio connector,
tokenizer assets, and custom Hugging Face model code. Load it with `trust_remote_code=True`.

For command-line and batch inference, multi-audio and multimodal in-context learning examples,
environment details, and fine-tuning instructions, see the
[SALMONN-2 GitHub repository](https://github.com/bytedance/SALMONN/tree/salmonn2).

## Results

SALMONN-2 achieves strong performance on three audio-language model (ALLM) benchmarks while using
18.2k hours of supervised audio-text training data. The table below shows the 8B model comparison.

| Model | Supervised audio-text data (h) | MMAU-Pro | MMAR | MMSU |
| --- | ---: | ---: | ---: | ---: |
| Qwen2.5-Omni | -- | 52.2 | 56.7 | 61.3 |
| Kimi-Audio | >13M | 56.6 | 60.8 | 54.7 |
| MiMo-Audio | >1M | 53.4 | 61.7 | 61.9 |
| AF-3 | >55k | 51.7 | 58.5 | 61.4 |
| MOSS-Audio | >1M | 57.5 | 64.4 | 66.4 |
| **SALMONN-2 8B** | **18.2k** | **58.5** | **64.5** | **69.5** |

## Minimal Hugging Face inference

The following example loads this repository directly with Hugging Face `transformers`; cloning or
installing the SALMONN-2 GitHub package is not required.

```bash
pip install "transformers>=4.57,<5" accelerate torch torchaudio
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "marcoyang/salmonn-2-8b-test"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()
processor.prepare_model(model)

inputs = processor(
    audios=["example.wav"],
    instruction="Please describe the audio.",
)
device = next(model.parameters()).device

with torch.inference_mode():
    output_ids = model.generate(
        **inputs.to(device),
        max_new_tokens=256,
        do_sample=False,
    )

print(processor.decode(output_ids[0]))
```

The processor loads and resamples audio, computes the SPEAR filterbank input, formats the chat
prompt, and returns all model inputs. The example uses bfloat16 and automatic device placement. A
CUDA GPU is recommended for practical inference.

### Contextual ASR

For text-only contextual words, pass a list of strings:

```python
inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=["howes", "wszelaki"],
)
```

To provide both the spelling and pronunciation of each contextual word, pair the text with a
context audio file:

```python
inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=[
        {"text": "howes", "audio": "howes.wav"},
        {"text": "wszelaki", "audio": "wszelaki.wav"},
    ],
)
```

The processor places all model-specific audio markers and contextual formatting automatically.

### Advanced prompt placement

For custom multi-audio layouts, use `formatted_prompt` with one `<audio>` marker per input file.
Audio files are matched to the markers from left to right:

```python
inputs = processor(
    audios=["main.wav", "example.wav"],
    formatted_prompt=(
        "<audio>Compare the main recording with this example: "
        "<audio>What do they have in common?"
    ),
)
```

The processor still loads the audio, computes filterbanks, validates marker alignment, applies the
chat template, and returns model-ready tensors.

## License

Apache License 2.0. See [LICENSE](./LICENSE).

## Citation

```bibtex
@inproceedings{yang2026spear,
  title     = {SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations},
  author    = {Yang, Xubo and Yang, Yuxuan and Jin, Ziyang and Cui, Zeyu and Wu, Wen and Li, Bo and Zhang, Chao and Woodland, Philip C.},
  booktitle = {Proceedings of the Forty-third International Conference on Machine Learning},
  year      = {2026}
}
```