sppandita85/bangalore-chat

Bangalore Chat Model

A small educational language model trained on Bangalore history (~10,000 tokens).

Model Description

This is a 2-layer transformer language model built from scratch for educational purposes. It demonstrates every component of an LLM: tokenization, embeddings (GloVe 300d), multi-head self-attention, feed-forward MLP blocks, and next-token prediction.

After pre-training on Bangalore history text, the model was fine-tuned on 50+ question-answer pairs to function as a simple Q&A assistant about Bangalore.

Architecture

Parameter	Value
Embedding dim	300 (GloVe 6B)
Transformer layers	2
Attention heads	6
MLP hidden dim	1,200
Max sequence length	128
Vocabulary size	50,257 (GPT-2)
Total parameters	~16M

Training

• Pre-training: Next-token prediction on ~10,000 tokens of Bangalore history
• Hardware: CPU only (Intel Mac Mini, 64GB RAM)
• Framework: PyTorch 2.x
• Embeddings: GloVe 6B 300d (pre-trained, loaded before training)

Intended Use

This model is educational only. It is not suitable for production use. Its purpose is to demonstrate how a language model is built from scratch.

How to Use

import torch
from transformers import GPT2Tokenizer

# Load tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("sppandita85/bangalore-chat")

# Load model (requires the BangaloreLM class from this repository)
# See the project README for full usage instructions.

Limitations

• Trained on only ~10,000 tokens — very limited knowledge
• May produce repetitive or incoherent text for out-of-distribution prompts
• Not suitable for any real-world application

Citation

Built as part of the Bangalore 10K LLM educational project.