youssefedweqd/working

🧠 Qwen2.5-1.5B - LoRA Fine-Tuned on Arabic Q&A 🕌

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-1.5B designed for Arabic Question Answering tasks. It was trained using the LLaMA-Factory framework on a custom curated dataset of Arabic Q&A pairs.

📚 Training Configuration

• Base Model: Qwen/Qwen2.5-1.5B
• Method: Supervised Fine-Tuning (SFT) with LoRA
• Framework: LLaMA-Factory
• Batch Size: 1 (gradient accumulation = 16)
• Epochs: 3
• Cutoff Length: 2048 tokens
• Learning Rate: 1e-4
• Scheduler: Cosine with warmup ratio 0.1
• Precision: bf16
• LoRA Rank: 64
• LoRA Target: all layers
• Eval Strategy: every 200 steps
• Eval Set Size: 3020 examples
• WandB Tracking: Enabled Run Link

📈 Evaluation (Epoch ~1.77)

• Eval Loss: 0.4321
• Samples/sec: 1.389
• Steps/sec: 0.695

🚀 Usage

You can use the model via transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Youssef/QWEN_Arabic_Q&A")
tokenizer = AutoTokenizer.from_pretrained("Youssef/QWEN_Arabic_Q&A")

prompt = "من هو مؤسس علم الجبر؟"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))


<|user|>
ما هي أركان الإسلام مع ذكر الحديث الذي يذكرها؟
<|assistant|>
أركان الإسلام خمسة، كما جاء في الحديث الصحيح:

عن عبد الله بن عمر رضي الله عنهما قال: قال رسول الله ﷺ: "بني الإسلام على خمس: شهادة أن لا إله إلا الله، وأن محمدًا رسول الله، وإقام الصلاة، وإيتاء الزكاة، وصوم رمضان، وحج البيت لمن استطاع إليه سبيلًا" (رواه البخاري ومسلم).

## 📉 Training Loss Over Epochs

| Epoch | Learning Rate | Loss   |
|-------|------------------------|--------|
| 0.16  | 5.39e-05               | 0.6304 |
| 0.18  | 5.88e-05               | 0.6179 |
| 0.19  | 6.37e-05               | 0.6042 |
| 0.21  | 6.86e-05               | 0.6138 |
| 0.22  | 7.35e-05               | 0.5940 |
| 0.24  | 7.84e-05               | 0.5838 |
| 0.25  | 8.33e-05               | 0.5842 |
| 0.26  | 8.82e-05               | 0.5786 |
| 0.28  | 9.31e-05               | 0.5713 |
| 0.65  | 9.60e-05               | 0.6122 |
| 0.71  | 9.45e-05               | 0.5809 |
| 0.77  | 9.29e-05               | 0.5446 |
| 0.82  | 9.10e-05               | 0.5339 |
| 0.88  | 8.90e-05               | 0.5296 |
| 0.94  | 8.67e-05               | 0.5176 |
| 1.00  | 8.43e-05               | 0.5104 |
| 1.06  | 8.17e-05               | 0.4685 |
| 1.12  | 7.90e-05               | 0.4730 |
| 1.18  | 7.62e-05               | 0.4679 |
| 1.24  | 7.32e-05               | 0.4541 |
| 1.30  | 7.01e-05               | 0.4576 |
| 1.35  | 6.69e-05               | 0.4472 |
| 1.41  | 6.36e-05               | 0.4427 |
| 1.47  | 6.03e-05               | 0.4395 |
| 1.53  | 5.69e-05               | 0.4305 |
| 1.59  | 5.35e-05               | 0.4280 |
| 1.65  | 5.01e-05               | 0.4251 |
| 1.71  | 4.67e-05               | 0.4188 |
| 1.77  | 4.33e-05               | 0.4177 |
| 1.83  | 3.99e-05               | 0.4128 |

**Evaluation Losses:**

- 📍 Epoch 1.18 → `0.4845`
- 📍 Epoch 1.77 → `0.4321`