swarajsonawane4/scifact-phi3-lora

SciFact Phi-3 Mini LoRA Adapter

LoRA adapter fine-tuned on the SciFact dataset for biomedical claim verification. Given a scientific claim and retrieved evidence, the model produces structured verdicts (SUPPORTED / REFUTED / INSUFFICIENT) with citations.

Use Case

This adapter sits on top of a RAG pipeline that retrieves biomedical evidence. The fine-tuned model generates grounded answers with explicit document citations, suitable for fact-checking applications.

Training Details

• Base model: microsoft/Phi-3-mini-4k-instruct (3.8B params)
• Method: LoRA (Parameter-Efficient Fine-Tuning)
• Rank: 8, Alpha: 16, Dropout: 0.05
• Target modules: qkv_proj, o_proj
• Quantization: 4-bit NF4 via bitsandbytes (for training)
• Training data: 759 examples from SciFact train split
• Evaluation data: 50 held-out examples
• Epochs: 5
• Effective batch size: 16
• Learning rate: 2e-4 (cosine decay)
• Hardware: Tesla T4 x2 (Kaggle)

Results

Epoch	Training Loss	Validation Loss
1	0.527	0.687
2	0.406	0.671
3	0.487	0.662
4	0.415	0.658
5	0.487	0.659

Generalization gap: 0.17 (healthy, no overfitting).

Usage

Load with PEFT on top of the base model:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE = "microsoft/Phi-3-mini-4k-instruct"
ADAPTER = "swarajsonawane4/scifact-phi3-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

Output Format

Verdict: SUPPORTED | REFUTED | INSUFFICIENT

Brief justification citing [D0], [D1], etc.

Author

Swaraj Sudhakar Sonawane - MSc. Digital Engineering, Bauhaus University Weimar