PersianSciQA-Qwen2.5-14B

A QLoRA fine-tuned adapter of Qwen/Qwen2.5-14B-Instruct for extractive question answering over Persian scientific texts. The model is trained on the safora/PersianSciQA-Extractive dataset using Supervised Fine-Tuning (SFT) with the TRL library.

Model Details

Model Description

This model is a LoRA adapter designed to answer questions about Persian scientific documents in an extractive manner. Given a scientific context (e.g., a research abstract) and a question in Persian, the model extracts the relevant answer directly from the provided context. When the context does not contain sufficient information to answer the question, the model is trained to respond with CANNOT_ANSWER.

• Developed by: safora
• Model type: Causal Language Model (LoRA Adapter)
• Language: Persian (Farsi)
• License: Apache 2.0
• Fine-tuned from: Qwen/Qwen2.5-14B-Instruct

Model Sources

• Repository: safora/PersianSciQA-Qwen2.5-14B
• Dataset: safora/PersianSciQA-Extractive
• Related Paper: PersianSciQA: A New Dataset for Bridging the Language Gap in Scientific Question Answering (RANLP 2025)

Uses

Direct Use

The model is intended for extractive question answering over Persian scientific texts. It can be used to build scientific information retrieval systems, academic research assistants, or Persian-language QA pipelines for the engineering and scientific domains.

Out-of-Scope Use

• General-purpose Persian chatbot or conversational use beyond scientific QA
• Answering questions without a provided context passage
• Non-Persian language tasks
• Medical, legal, or safety-critical decision-making

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Quantization config (same as training)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-14B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Load the LoRA adapter
model = PeftModel.from_pretrained(base_model, "safora/PersianSciQA-Qwen2.5-14B")
tokenizer = AutoTokenizer.from_pretrained("safora/PersianSciQA-Qwen2.5-14B")

# Build the prompt
context = "متن علمی فارسی شما اینجا قرار می‌گیرد..."
question = "سوال شما اینجا قرار می‌گیرد..."

prompt = (
    'شما یک دستیار متخصص در زمینه اسناد علمی هستید. وظیفه شما این است که به سوال پرسیده شده، '
    '**فقط و فقط** بر اساس متن زمینه (Context) ارائه شده پاسخ دهید. پاسخ شما باید دقیق و خلاصه باشد.\n\n'
    '**دستورالعمل مهم:** اگر اطلاعات لازم برای پاسخ دادن به سوال در متن زمینه وجود ندارد، '
    'باید **دقیقا** عبارت "CANNOT_ANSWER" را به عنوان پاسخ بنویسید و هیچ توضیح اضافه‌ای ندهید.\n\n'
    f'**زمینه (Context):**\n---\n{context}\n---\n\n'
    f'**سوال (Question):**\n{question}\n\n'
    '**پاسخ (Answer):**'
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

answer = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(answer)

Training Details

Training Data

The model was fine-tuned on the safora/PersianSciQA-Extractive dataset. The PersianSciQA dataset is a large-scale resource of question–answer snippet pairs derived from Persian scientific engineering abstracts sourced from IranDoc's Ganj repository. Each sample contains a scientific context, a question, and an extractive answer.

Training Procedure

The model was fine-tuned using QLoRA (Quantized Low-Rank Adaptation) with the TRL SFTTrainer.

Quantization (QLoRA)

Parameter	Value
Quantization	4-bit (NF4)
Compute dtype	bfloat16
Double quantization	No

LoRA Hyperparameters

Parameter	Value
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Hyperparameters

Parameter	Value
Epochs	1
Per-device train batch size	1
Gradient accumulation steps	8
Effective batch size	8
Learning rate	2e-5
LR scheduler	Cosine
Warmup ratio	0.03
Optimizer	paged_adamw_32bit
Max gradient norm	1.0
Precision	bf16
Max sequence length	1024

Framework Versions

• PEFT: 0.17.1
• Transformers: (HuggingFace Transformers)
• TRL: (SFTTrainer)
• PyTorch: (with CUDA / bfloat16 support)
• bitsandbytes: (4-bit quantization)

Evaluation

The model was evaluated on the validation split during training (evaluation at each epoch). A held-out test split is available in the dataset for final evaluation.

Bias, Risks, and Limitations

• The model is specifically designed for Persian scientific texts in the engineering domain. Performance on other domains or languages is not guaranteed.
• Answers are only as reliable as the provided context passage — the model does not have access to external knowledge during inference.
• The model may occasionally fail to produce CANNOT_ANSWER when the context is insufficient, or may extract incomplete answers.
• As a fine-tuned LLM, it may inherit biases present in the base Qwen2.5-14B-Instruct model or in the training dataset.

Citation

If you use this model, please cite the PersianSciQA dataset paper:

@inproceedings{jolfaei2025persiansciqa,
  title={PersianSciQA: A New Dataset for Bridging the Language Gap in Scientific Question Answering},
  author={Jolfaei, Safoura Aghadavoud and Mohebi, Azadeh and Hemmat, Zahra},
  booktitle={Proceedings of Recent Advances in Natural Language Processing (RANLP)},
  pages={32--37},
  year={2025},
  address={Varna, Bulgaria}
}