🦙 MicroLlama-134M-Instruct

MicroLlama-134M-Instruct is a custom-trained Small Language Model (SLM) created by Abhiray (Jay). Built using a scaled-down Llama architecture, this model is designed to be highly efficient, lightweight, and can try to conversational instruction-following.

🧠 Model Details

• Parameters: 134 Million
• Architecture: Custom Llama (12 Layers, 12 Heads, 768 Embedding Dimension, 2048 Intermediate Size)
• Context Length: 1024 Tokens
• Precision: bfloat16
• Tokenizer: TinyLlama-1.1B-Chat-v1.0 (32,000 Vocab)

🏗️ Training Pipeline

This model was trained entirely from scratch in three distinct phases:

1. Phase 1 & 2: Continuous Pre-Training (CPT): Trained on ~9 Billion tokens of high-quality web, educational, and narrative data.
2. Phase 3: Supervised Fine-Tuning (SFT): Fine-tuned for 250 high-precision steps using -100 target masking.

💻 How to Use

The model strictly follows the ChatML-style template used during its SFT phase. For optimal performance, a generation temperature between 0.3 and 0.5 with a gentle repetition_penalty (e.g., 1.05) is recommended.

Prompt Format:

<|system|>
You are a highly capable, friendly, and helpful AI assistant.</s>
<|user|>
What is the core temperature of the Sun?</s>
<|assistant|>