🚀 PyBlissa-Coder-50M

!! BENCHMARK ON HumaEval DATASET: 10.4% !!

A 50M-parameter Python code generation model trained from scratch on a single RTX 5080. Built as part of the PRIME lineup of small, locally-runnable AI systems.

Despite its size, PyBlissa punches well above its weight on Python instruction-following tasks. Trained near-Chinchilla optimal (~13 tokens/parameter) for maximum capacity utilization.

This model punched a solid 10.4% score in OpenAI's HumanEval dataset, which is an amazing number for this model's size! However, this model technically can generate bad outputs. You'd need to tweak the temperature. But that's a rare case!

Stats

Parameters	50.2M
Architecture	Decoder-only transformer (GPT-2 style)
Context length	1024 tokens
Vocab size	16,000 (custom ByteLevel BPE)
Train tokens	166M
Final val loss	0.474
Training time	73 minutes (RTX 5080)

Architecture

d_model:      640
n_layer:      8
n_head:       8
d_ff:         2560
block_size:   1024
tied embeddings, pre-LN, no bias, GELU MLP, SDPA attention

Training data

Two-source code-instruction corpus, 425k samples → 166M tokens after BPE tokenization:

• nvidia/OpenCodeInstruct — 400k high-quality instruction-code pairs
• flytech/python-codes-25k — 25k Python-focused instruction-code pairs

Trained for 4 epochs with cosine LR schedule (3e-4 → 3e-5), bf16 autocast, batch size 20.

Prompt format

Trained on a strict prefix structure:

PROMPT: <your instruction>
CODE:
<generated code>

Anything else is out-of-distribution. The Modelfile in this repo handles the formatting automatically.

Usage — Ollama (recommended)

ollama run hf.co/Rohanify/PyBlissa-Coder-50M:F32

Or pull the GGUF directly and create locally:

ollama create pyblissa-coder -f Modelfile
ollama run pyblissa-coder "write a function to merge two sorted lists"

Usage — llama.cpp

./llama-cli -m PyBlissa-Coder-50M-F32.gguf \
  -p "PROMPT: write a function to reverse a string\nCODE:\n" \
  --temp 0.8 --top-k 50 --top-p 0.95 -n 256

Files

File	Purpose
PyBlissa-Coder-50M-F32.gguf	Full-precision GGUF weights
Modelfile	Ollama config (prompt template, stop tokens, sampling)
tokenizer.json	Custom 16k BPE tokenizer

Limitations

• Mostly Python — other languages weren't that much in training data
• 1024-token context — longer programs get truncated
• Small flytech subset (~6% of training data) contains code with unescaped quote bugs; the model occasionally inherits this pattern
• No safety tuning, no RLHF — base model only

Acknowledgments

Datasets by NVIDIA and flytech. Built using the nanoGPT-style training recipe with custom tokenization. Tooling: PyTorch, HuggingFace tokenizers, llama.cpp for GGUF conversion.

---

Made by Rohan. Also known as ElectroPlayin on YouTube