QED-Base-v2
QED-Base-v2 is a ~107M parameter causal language model pretrained from scratch by Qarvexium. It is a base model — it has not been instruction-tuned or aligned for chat, and will continue text rather than follow instructions or hold a conversation.
Model Details
- Developed by: Qarvexium
- Model type: Decoder-only causal language base model
- Language: English
- License: MIT
- Tokenizer: QED-B1 tokenizer (SentencePiece BPE, 48,000 vocabulary)
Architecture
| Component | Value |
|---|---|
| Tokenizer | QED-B1 tokenizer |
| Vocabulary size | 48,000 |
| Model type | Decoder-only Transformer |
| Parameters | ~107M |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 |
| KV heads | 4 |
| Attention | GQA |
| Intermediate FFN size | 1792 |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Position encoding | RoPE |
| Context length | 2048 |
| RoPE theta | 10000 |
Weight-tied embeddings/LM head.
Uses
Direct Use
As a base model, QED-Base-v2 is intended for:
- Text completion / continuation
- Research on small-scale language model pretraining
- A starting checkpoint for further fine-tuning (instruction tuning, domain adaptation, etc.)
Out-of-Scope Use
This model has not been instruction-tuned, RLHF'd, or safety-aligned. It should not be deployed directly in a chat or assistant product, or in any application where reliable instruction-following or content moderation is required, without further fine-tuning and evaluation.
Bias, Risks, and Limitations
QED-Base-v2 was trained on a large web-crawled corpus and will reflect the biases, inaccuracies, and occasionally toxic content present in that data. As a base model it has no built-in refusal behavior or safety tuning — outputs should be filtered/evaluated before use in any user-facing setting. At ~107M parameters, factual accuracy and reasoning ability are limited compared to larger models.
How to Get Started
from infer import load_model, load_tokenizer, run
model = load_model("QED-Base-v2.pt")
tokenizer = load_tokenizer("tok.model")
text = run("Once upon a time", model, tokenizer, max_new_tokens=100)
print(text)
License
MIT License