genaforvena/finnegans-fake-bpe4096

Finnegans Fake — BPE-4096

The same book through a 4096-token BPE vocabulary trained on the book itself.

On the book. These weights are trained on the full text of Finnegans Wake. Joyce died in 1941, so the book is public domain in Ireland, the UK and the EU — in the United States it is not, until 2035. Whether trained weights are a derivative work of their training text is unsettled either way. The corpus is not redistributed with the code; the code is CC0 and you supply the book.

What it is

architecture	GPT-2 (6 layers, 6 heads, 384 embd)
parameters	12,318,720
vocabulary	4096
context	256 tokens
trained	6,000 iters, batch 32, lr 0.0006, dropout 0.2
best val loss	5.8768
end-of-run val loss	7.7704

This repo carries the best-val checkpoint, not the end of the run.

That distinction is load-bearing here: the run ended 1.89 nats above its own best, so the last checkpoint is meaningfully worse than this one.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("finnegans-fake-bpe4096")
model = AutoModelForCausalLM.from_pretrained("finnegans-fake-bpe4096")

ids = tok("riverrun, past Eve and Adam's,", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True,
                     temperature=0.9, top_k=100, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Sampled at pack time, seed 0, so this is reproducible rather than curated:

riverrun, past Eve and Adam's, tuted, trun's billy, t he's moth, like a pair her ciel, the suddin's in her trave, with her grimmt or she spit her bidly, her she's plink her jawd she a tickle jash she slave her crom, how she's a shine her she was she g

One thing to know about the tokenizer

It is a byte-level BPE with add_prefix_space: true, so decoding inserts a single leading space that was not in your prompt:

tok.decode(tok("riverrun").input_ids)   # -> ' riverrun'

Round-trip is otherwise exact — verified on 20k characters of the training corpus, byte for byte, once that one space is accounted for. Prompt with a leading space, or strip one from the output; do not go looking for a lossy character.

The corpus this measures

running words	224,527
distinct word types	58,725
types occurring exactly once	46,599 (79.4%)
text covered by types seen >=5 times	69.8%
distinct characters	105

Four in five word types are hapax legomena, which settles tokenisation before any training: a word-level vocabulary is not merely coarse here, it is impossible — most types would carry one example, and every unseen word becomes <unk>, so the model could never coin one. Coining is the only thing worth wanting from it.

A bug worth keeping in the card

Every loss figure this project first published was produced by a broken objective. batch() returned nanoGPT-style pre-shifted labels while transformers shifts labels itself, so the shift happened twice and every run learned to predict token t+2 from position t. It never raised: training ran, loss fell, the curves looked plausible, and the inflated perplexity supported a confident story about the Wake being statistically incompressible. That story was a property of the bug. Only the generated text exposed it, by coming out looking like every second character had been deleted. After the fix, 400 steps beat the 6000 broken ones.

The figures in this card are post-fix.

What it is not

It is not good, and it is not trying to be. The aim was to find out what a language model does when the only language it has ever seen is one book — whether anything resembling English survives, and whether the machine can coin words the way Joyce did rather than quote the ones he already coined. Do not use it for anything.

Code: https://github.com/genaforvena/finnegans-fake — CC0.