Whisper Small Fine-Tuned β€” Yoruba

This model is a fine-tuned version of openai/whisper-small for Yoruba speech recognition. It was trained on Common Voice 25.0 (yo) augmented with Google Fleurs (yo_ng) for a total of 3,718 training samples.

WER (raw, diacritics preserved):

  • Test set (1,071 samples) : 73.67 %
  • Validation set (975 samples) : 71.50 %

The WER is computed without any text normalisation in order to preserve Yoruba diacritics (tone marks, sub-dots, etc.). Lowercasing or stripping diacritics would inflate the reported score, but the model genuinely generates them.

Model description

Whisper Small is a transformer-based encoder-decoder model for automatic speech recognition (ASR), pre-trained on 680k hours of multilingual data. It uses 80-channel log-mel spectrograms as input and outputs a sequence of text tokens via autoregressive decoding with cross-attention to the encoder.

This fine-tuned version adapts the model specifically to Yoruba, a Niger-Congo language spoken in Nigeria, Benin and Togo. The model generates Yoruba text with full diacritic marks (à, è, ẹ, í, ò, ọ, ú, ṣ, etc.).

Intended uses & limitations

Intended uses:

  • Transcribing Yoruba speech from audio recordings
  • ASR research for low-resource African languages
  • Benchmarking diacritic-preserving ASR systems

Limitations:

  • Trained on ~4.3 epochs only (500 steps). More training may improve WER.
  • Training data is mostly read-speech from Common Voice; performance on conversational/spontaneous Yoruba may be lower.
  • The model retains Whisper's 30-second audio window.
  • Diacritic coverage may be imperfect for some Yoruba dialects.

Training and evaluation data

  • Common Voice 25.0 (yo) β€” 1,422 train / 975 validation / 1,071 test samples
  • Google Fleurs (yo_ng, raw_transcription field to preserve diacritics) β€” 2,339 train samples (+43 removed by the >30 s filter)

Only clips shorter than 30 seconds were kept. Combined training set: 3,718 samples.

Training procedure

Hardware

  • 2 Γ— NVIDIA Tesla T4 (15.6 GB each)
  • Distributed Data Parallel (DDP) via accelerate launch
  • Mixed precision: fp16 (Native AMP)
  • Total training time: ~26 minutes (1,605 seconds)

Training hyperparameters

  • learning_rate: 1e-05
  • train_batch_size (per device): 16
  • eval_batch_size (per device): 16
  • gradient_accumulation_steps: 1
  • total_train_batch_size (effective): 32
  • total_eval_batch_size (effective): 32
  • seed: 42
  • optimizer: AdamW (torch fused) with betas=(0.9, 0.999) and epsilon=1e-08
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_steps: 50
  • max_steps: 500
  • gradient_checkpointing: True
  • eval_strategy: steps (every 500 steps)
  • save_strategy: steps (every 500 steps, max 2 checkpoints)
  • generation_max_length: 225
  • predict_with_generate: True
  • mixed_precision_training: Native AMP (fp16)

Training results

Training Loss Epoch Step Validation Loss Validation WER
0.4559 4.2735 500 0.7070 71.50 %

Qualitative examples (test set, seed=42)

PrΓ©diction RΓ©fΓ©rence WER
AΜ€kọ̀wé αΊΉgbẹ́ wá ní, aΜ€wọn ọmọ αΊΉgbẹ́ ọ̀mú ti pú t Akọ̀wΓ© αΊΉgbẹ́ wa nΓ­ Γ wọn ọmọ αΊΉgbẹ́ ọhΓΊn ti kΓΊ tΓ‘n. 63.6 %
LΓ³Γ²Γ³tọ́ nΓ­ Γ wọn alΓ‘boyΓΊ mΓ­rΓ n ma ń kan rΓ‘. LΓ³Γ²Γ³tọ́ ni Γ wọn alΓ‘boyΓΊn mΓ­rΓ n mΓ‘a Ε„ kanra 75.0 %
ỌkΓΉnrin Γ ti obΓ¬nrin tΓ­ kΓ² nΓ­ Γ¬bΓ‘lΓ²pọ̀ kΓ² lΓ© ibi mọ́ ỌkΓΉnrin Γ ti obΓ¬nrin tΓ­ kΓ² nΓ­ Γ¬bΓ‘lΓ²pọ̀ koΜ€ leΜ€ bímọ 40.0 %

Framework versions

  • Transformers 5.12.1
  • PyTorch 2.10.0+cu128
  • Datasets 5.0.0
  • Tokenizers 0.22.2
  • Accelerate 1.14.0
Downloads last month
14
Safetensors
Model size
0.2B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for KalineZephyr/whisper-small-yoruba-finetuned

Finetuned
(3,861)
this model

Dataset used to train KalineZephyr/whisper-small-yoruba-finetuned

Space using KalineZephyr/whisper-small-yoruba-finetuned 1

Evaluation results