neuron-to-gpt
A learning journal that starts at one neuron and ends at a trained character-level GPT, with the architecture's gains measured stage by stage.
Know the machine you evaluate
Evaluating language models for a living without ever having built one leaves a gap. This journal closes it the slow way: a neuron, a loss, a gradient, then an autograd engine written with nothing but the standard library, and only then PyTorch.
From week three each new layer is rebuilt by hand before the framework version is trusted: the bigram counts before the neural bigram, manual backpropagation before backward(), a custom BatchNorm before LayerNorm, causal averaging before attention. The Week 8 model was written before the finished reference was consulted.
Eight weeks, then a measured comparison
The six-stage comparison holds the corpus, split, seed and iteration budget fixed and adds one idea at a time. Each step improved in this run; the repository says plainly that improvement is not guaranteed for every configuration.
What the trained models reached
Nats per character from bigram to the completed block, at a fixed CPU budget.
Final validation NLL after 5,000 updates; best sampled checkpoint 1.4657 at step 4,500.
A 1.9M-character Nâzım Hikmet collection with a 118-character vocabulary. Not a ranking against English.
An optional panel loads the trained checkpoints and generates text next to the recorded evidence.
The Turkish corpus was prepared from three user-selected PDFs with layout-aware extraction and OCR recovery of 243 pages. Raw texts, PDFs and checkpoints stay out of Git; only metrics, loss histories, plots and samples are published.
What 4.8 million parameters can and cannot do
An unedited excerpt generated by the Shakespeare model, not a quotation:
KING RICHARD II: Shall I, be made to be exceeding?QUEEN ELIZABETH: My hard will I be?
The model has learned speaker labels, line structure and plausible vocabulary. It has not learned meaning. A low character loss does not guarantee coherent writing, and the repository states that next to the samples.
Educational scale, and stated as such. The larger runs change width, depth, context, training budget and precision together, so they are not an ablation; the six-stage CPU table is the controlled comparison, and it is not parameter-matched. The reported GPU losses were validated for internal consistency against the returned histories, not re-evaluated independently. The sequence follows Andrej Karpathy's lecture series.