WhisperDRZ

WhisperDRZ is a speaker-aware automatic speech recognition model. It transcribes audio into text with word-level timestamps, per-line speaker tags, and non-speech event tags ([laugh], [breath], ...). It is a Whisper-style encoder-decoder model and handles long audio by chunking and stitching internally.

• 🎙️ Try it in the browser — no install: fluxions.ai/transcribe
• 💻 Code (inference): https://github.com/fluxions-ai/whisperdrz
• 📖 Full write-up — approach, honest results, what didn't work: see the repo's writeup.md

Usage

pip install git+https://github.com/fluxions-ai/whisperdrz

Requires Python 3.12+ and FFmpeg. Runs on a CUDA GPU or on CPU. Weights download automatically from this repo on first use.

Command line

whisperdrz audio.wav                          # defaults: this model, --lang en
whisperdrz audio.wav --output_format json > out.json
whisperdrz audio.wav --lang auto              # auto-detect language

Python

import whisperdrz
from whisperdrz.audio import load_audio, SAMPLE_RATE

transcriber = whisperdrz.load_model("whisperdrz-large-v3.safetensors", lang="en")

audio, _ = load_audio("audio.wav", sample_rate=SAMPLE_RATE)
result = transcriber.transcribe(audio.mean(0))  # mono, 16 kHz

print(result.text)        # speaker-tagged text with timestamps
print(result.segments)    # list of {speaker, start, end, text}

You can also point load_model at this repo id (fluxions/whisperdrz) directly.

Output format

Each line begins with a speaker tag. Timed words and tags are wrapped in a start/end timestamp pair; the first and last word of each line are always timed:

[0] <|0.00|>Hello<|0.45|> there <|0.80|>world.<|1.10|>
[1] <|1.20|>Hi<|1.40|> <|1.45|>[laugh]<|1.60|> <|1.70|>there.<|1.95|>

• [0], [1], ... are speaker IDs; [c] marks crowd/ambient.
• <|t|> are timestamps in seconds (two decimals), always in a pair wrapping a word or tag.
• [laugh], [breath], and similar are non-speech event tags.

Evaluation

Measured on this checkpoint:

Benchmark	WER	DER (miss / FA / conf)	cpWER / tcpWER
ESB (English ASR, 1000 utts)	9.6% macro / 5.9% micro	—	—
Internal conversational (26 clips)	11.1%	— (WDER 33%)	46% cpWER
VoxConverse dev (216, overlap-heavy)	—	26.3% (3.1 / 15.8 / 7.5)	—
CALLHOME eng (140, 2-spk telephone)	—	38.6% (5.9 / 14.0 / 18.7)	—
AMI test (16 meetings, Mix-Headset)	22.8%	50.6% (10.6 / 28.9 / 11.1)	72% / 84%

DER is at the standard 0.25s collar, scoring overlapping speech. AMI cpWER/tcpWER and the 22.8% WER use oracle speaker count.

WhisperDRZ is an ASR-first model. Transcription is strong across the board, but diarization trails purpose-built systems (~10–25% DER) — speaker confusion dominates on long, overlapping multi-party audio.

No VAD. It transcribes over silence and music, so false alarm is a large part of DER. Gating to detected speech before scoring recovers much of it:

Benchmark	DER (ungated)	+ silero VAD	+ oracle VAD
VoxConverse dev	26.3%	24.0%	20.9%
CALLHOME eng	38.6%	34.2%	28.6%
AMI test	50.6%	40.6%	31.1%

See the write-up in the code repo for full analysis (timing, non-speech events, multilingual).

License

MIT.