chhatramani/medqa-gemma-3nE4B-4bit

MedQA-Gemma-3n-E4B-4bit

A 4-bit quantized Gemma-3n-E4B model fine-tuned on medical Q&A data using Unsloth for efficient training.

Model Details

Overview

• Model type: Fine-tuned Gemma-3n-E4B (4-bit QLoRA)
• Purpose: Medical question answering
• Training approach: Instruction fine-tuning
• Dataset: 1,000 samples from MIRIAD-4.4M

Specifications

Feature	Value
Base Model	google/gemma-3n-E4B-it
Quantization	4-bit (QLoRA)
Trainable Parameters	19,210,240 (0.24% of total)
Sequence Length	1024 tokens
License	CC-BY-SA-4.0

Training Information

Hyperparameters

{
    "per_device_batch_size": 2,
    "gradient_accumulation_steps": 8,
    "effective_batch_size": 16,
    "num_epochs": 5,
    "total_steps": 300,
    "learning_rate": 3e-5,
    "loRA_rank": 16,
    "loRA_alpha": 32,
    "optimizer": "adamw_8bit",
    "lr_scheduler": "cosine",
    "warmup_steps": 50,
    "weight_decay": 0.01,
    "max_seq_length": 1024
}

Evaluation Results

Metric	Value
BLEU-4	0.42
ROUGE-L	0.58
BERTScore-F1	0.76
Perplexity	12.34

Note: Evaluated on 100-sample test set

Limitations

• Scope: Trained on only 1,000 examples - not suitable for clinical use
• Knowledge cutoff: Inherits base model's knowledge limitations
• Precision: 4-bit quantization may affect some reasoning tasks
• Bias: May reflect biases in both base model and training data

Ethical Considerations

• Intended Use: Research/educational purposes only
• Not for: Clinical decision making or medical advice
• Bias Mitigation: Users should apply additional filtering for sensitive applications

Citation

@misc{medqa-gemma-3nE4B-4bit,
  author = {Chhatramani, YourName},
  title = {MedQA-Gemma-3n-E4B-4bit: Medical Q&A Fine-tuned Model},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/chhatramani/medqa-gemma-3nE4B-4bit}}
}