File size: 3,423 Bytes
27b78ee 97fd364 f2219b2 35bdb63 372bfb5 43d1d2f f2219b2 27b78ee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | ---
license: mit
language:
- en
tags:
- fish
- character
- tiny-llm
- text-generation
- from-scratch
pipeline_tag: text-generation
---
<p align="center">
<img src="assets/guppy.png" alt="GuppyLM" width="300"/>
</p>
<p align="center">
<a href="https://github.com/arman-bd/guppylm"><img src="https://img.shields.io/badge/GitHub-guppylm-181717?logo=github" alt="GitHub"/></a>
<a href="https://huggingface.co/datasets/arman-bd/guppylm-60k-generic"><img src="https://img.shields.io/badge/🤗_Dataset-guppylm--60k-blue" alt="Dataset"/></a>
<a href="https://colab.research.google.com/github/arman-bd/guppylm/blob/main/use_guppylm.ipynb"><img src="https://img.shields.io/badge/Open_in-Colab-F9AB00?logo=googlecolab" alt="Colab"/></a>
<br/>
<a href="https://www.linkedin.com/pulse/build-your-own-language-model-5-minutes-i-made-mine-hossain--supif/"><img src="https://img.shields.io/badge/Article-LinkedIn-0A66C2?logo=linkedin" alt="LinkedIn Article"/></a>
<a href="https://arman-bd.medium.com/build-your-own-llm-in-5-minutes-i-made-mine-talk-like-a-fish-e20c338a3d14"><img src="https://img.shields.io/badge/Article-Medium-000000?logo=medium" alt="Medium Article"/></a>
<br/><br/>
<a href="https://arman-bd.github.io/guppylm/"><img src="https://img.shields.io/badge/Try_in-Browser-64ffda?style=for-the-badge&logo=webassembly&logoColor=white" alt="Browser Demo"/></a>
</p>
# GuppyLM — 9M Parameter Fish Chat Model
A ~9M parameter LLM trained from scratch that talks like a small fish named Guppy.
This project exists to show that training your own language model is not magic. One Colab notebook, 5 minutes, and you have a working LLM built from scratch.
## Example
```
You> hi guppy
Guppy> hi there. i just found a nice spot near the rock. the temperature feels nice.
You> are you hungry
Guppy> yes. always yes. i will swim to the top right now. i promise to eat all of it.
You> what is the meaning of life
Guppy> food. the answer is always food.
You> tell me a joke
Guppy> what did the fish say when it hit the wall. dam.
You> goodnight guppy
Guppy> ok sleep time. i was following a bubble but now i'll stop. goodnight tank. goodnight water.
```
## Architecture
| | |
|---|---|
| **Parameters** | 8.7M |
| **Type** | Vanilla transformer (from scratch) |
| **Layers** | 6 |
| **Hidden dim** | 384 |
| **Heads** | 6 |
| **FFN** | 768 (ReLU) |
| **Vocab** | 4,096 (BPE) |
| **Max sequence** | 128 tokens |
| **Norm** | LayerNorm |
| **Position** | Learned embeddings |
| **LM head** | Weight-tied with embeddings |
No GQA, no RoPE, no SwiGLU, no early exit. As simple as it gets.
## Training
- **Data:** 60K single-turn synthetic conversations across 60 topics
- **Steps:** 10,000
- **Optimizer:** AdamW (cosine LR schedule)
- **Hardware:** T4 GPU (~5 min)
- **No system prompt** — personality is baked into the weights
## Usage
```python
from inference import GuppyInference
engine = GuppyInference('checkpoints/best_model.pt', 'data/tokenizer.json')
r = engine.chat_completion([{'role': 'user', 'content': 'hi guppy'}])
print(r['choices'][0]['message']['content'])
# hi there. i just found a nice spot near the rock.
```
## Links
- **Repo:** [github.com/arman-bd/guppylm](https://github.com/arman-bd/guppylm)
- **Dataset:** [huggingface.co/datasets/arman-bd/guppylm-60k-generic](https://huggingface.co/datasets/arman-bd/guppylm-60k-generic)
## License
MIT
|