gvij/minicpm-av-music-avqa

MiniCPM-AV Music-AVQA Model

This is a fine-tuned MiniCPM-AV model trained on the MUSIC-AVQA-v2.0 dataset for audio-visual question answering.

Model Details

• Base Model: openbmb/MiniCPM-V
• Audio Encoder: UsefulSensors/moonshine-tiny
• Training Dataset: DraculaDragon/MUSIC-AVQA-v2.0
• Training Config:
  • Epochs: 3
  • Batch Size: 64
  • Learning Rate: 2e-5
  • LoRA Rank: 8
  • LoRA Alpha: 16
  • Mixed Precision: bf16
  • Best Validation Loss: 0.1525

Files

• audio_components.pt: Audio projector, compressor, and modality embeddings
• adapter_config.json: LoRA adapter configuration
• adapter_model.safetensors: LoRA adapter weights (13.6M parameters)

Usage

from src.modeling_minicpm_av import MiniCPMAV, MiniCPMAVConfig
import torch

# Load model
config = MiniCPMAVConfig()
model = MiniCPMAV(config=config)

# Load pretrained weights
model.load_pretrained("gvij/minicpm-av-music-avqa")
model.eval()

# Prepare inputs
audio = torch.randn(16000)  # 1 second of audio
image = PIL.Image.open('image.jpg')
question = "What instrument is playing?"

# Generate answer
answer = model.generate_with_audio(
    images=image,
    audio=audio,
    question=question
)
print(answer)

Citation

@misc{minicpm-av,
  title={MiniCPM-AV: Audio-Visual Question Answering with MiniCPM-V},
  author={Gaurav Vij},
  year={2025}
}