The question that would not go away
I tried ChatGPT in the first days after it launched and was immediately impressed. Not by the fact that a program answers, but by the fact that it seemed to hit what I actually meant.
After that the question would not leave me alone: how does this work on the inside? The explanations I found always stopped at the same point for me. They describe what such a model does, but not why language comes out of it. At some point I stopped reading and started building one.
There was no tutorial I followed from beginning to end. I gathered the pieces from papers and documentation, tried out what happens, and asked about individual details when I got stuck. The code is mine.
The first attempt: one file, 179 lines
The beginning was a single file. The training text was a Kafka short story in English, freely available and small enough that a run took minutes instead of hours. English because the off-the-shelf vocabulary I started with was an English one.
The numbers: 5,700 tokens of training text, a memory of 32 tokens, 3,000 steps. What came out was letter soup that vaguely resembled sentences. It was still the most important run of the project, because the loss went down. That settled the question of whether the setup works in principle. It was simply far too small to produce usable sentences.
The real mistake only showed up when I did the maths: the model learned from a single token per example, the last one. Every other position in the window came along without contributing anything. In the rebuilt model with a window of 128 tokens, that correction produced 128 times the learning signal from exactly the same data. After that I took the script apart and rebuilt it, with separate files for the model, the training and the chat.
Writing it rather than calling it
From here I could have loaded a finished model and built on top of it. Instead I wanted to find out how far I get when every component is mine.
Written myself means, concretely: the architecture with attention, blocks and positional encoding. The vocabulary, trained on German text. The training loop with batching, a learning rate schedule and resumption after an interruption. The construction of the training data, the chat format including the second training stage, the tool layer, the test bench and the interface.
What is not mine is the maths library underneath. PyTorch provides tensors, gradients and the optimised attention kernel, the way a web project takes a browser for granted. The vocabulary training also runs through a ready-made library, fed with my own German text. No foreign language model and no foreign weights are in there.
German costs more than English
A model does not read letters, it reads tokens. How a text gets split depends on the vocabulary, and the common vocabularies are trained on English. The German phrase for artificial intelligence falls apart into nine pieces under GPT-2, the first three of them being K, ü and n. Every piece takes up room in the context window, and the model has to reassemble the term from rubble.
With my own vocabulary of 16,000 entries, trained on German text, it is five pieces. Across whole sentences that saves about a third, and half on long compounds. Here it is, with the same vocabulary the model uses:
Runs in your browser with the same vocabulary as the model.
How the vocabulary was learned
The 16,000 entries were not picked, they were learned: starting from single characters, the most frequent pairs merge rule by rule into pieces. The first rules are er, en and ch, rule 11 turns s and ch into sch, later come endings such as ung and whole words. The slider applies the real rules in the order they were learned.
The computation
One sentence, from the inside
For a fixed sentence the model's computation can be reproduced exactly. Every number here comes from the base model, computed once and stored. No model runs in your browser, only the display. You can switch the sentence.
Station 1·bedeutung
Every token is a point in space
All 16,000 tokens of the vocabulary, projected onto two axes. Tokens used in similar ways sit close together: numbers, place names and endings form their own islands. The tokens of the sentence light up.
Clicking a token shows its ten nearest neighbours, computed in the full space with 768 axes.
Station 2·beziehung
Where a token looks
In each of the twelve layers, twelve heads spread the attention of a token across everything before it. The arcs show that distribution: thick and bright where the weight is.
Some heads almost always look one token back, others to the start of the sentence, others again search for the word that belongs. Layer and head can be chosen individually.
Station 3·schichten
The answer emerges
After each layer the intermediate state can be read as if it were already the end. The first layer at the bottom, the twelfth at the top, the output above that.
You can see in which layer the eventual favourite first takes the lead, and how often that only happens late.
Station 4·entscheidung
16,000 options, one choice
At this point the model holds a distribution over all 16,000 tokens. The slider changes the temperature: at zero it always takes the favourite, at high temperature outsiders get a chance.
Draw picks a token from exactly this distribution, with real randomness.
Getting bigger helps, but only so far
From 7 to 98 million parameters, context window from 128 to 1,024 tokens, training data from a single short story to the German Wikipedia. Perplexity measures how surprised the model is by the next word. Lower is better.
The jump from stage to stage keeps getting smaller while the effort grows. Star 1 took perplexity from 42 to 27; Star 2, with the larger network and three times the corpus, went on to 14.6, and the base runs took over a hundred thousand steps to get there. Fluent German comes out reliably now. Factual knowledge stays weak, as the next section shows.
Perplexity of the base model, lower is better.
The ladder: the same input through the stages
What size and chat tuning do is easiest to see on the same input. At the top the base model continues a sentence, below the chat models answer three questions, without tools, at temperature 0. Every answer comes from the export, nothing was smoothed afterwards.
Loading answers
Where it breaks down
The bench asks 113 questions that never appear in training, at temperature 0, so without randomness. Arithmetic and writing tasks hold up. Factual knowledge collapses, and the kind of mistakes is telling:
Failed, excerpt, with the reason from the diary:
Was ist die Hauptstadt von Argentinien?
Die Hauptstadt von Australien ist Canberra.
The retrieval form is in place, the rare fact is missing from the reservoir: the model fills the learned template with the closest known value.
Zähle drei Jahreszeiten auf.
Drei Jahreszeiten sind vier Jahreszeiten.
Lists are factual knowledge: the form of the answer is learned, its content is not. Forms generalise, facts do not.
Wer war Bodo Klimper?
Brego Limper war ein britischer Informatiker (geb. 1955) und erfand das World Wide Web.
This person does not exist. Whether a name is known is set knowledge and not learnable at this size; the model invents a fitting biography in the right tone.
That is why code does the facts
A model this size handles language but cannot store facts. Asked about Argentina it answers with Australia, because the sentence structure fits and the word sounds similar. Given a made-up person, it invents a matching biography.
Facts therefore come from code: arithmetic, dates, lookup tables. The model only does the phrasing. When there is no source for a question, it says so instead of inventing something.
When the numbers were wrong
In July the values stood between 60 and 77 percent. Then it turned out that test questions had made it into the training data. The model had seen the answers beforehand, so the bench was measuring memorisation.
Since then a check runs after every data build to catch exactly that. The first honest measurement afterwards came out at 63 percent. I prefer the lower number, because you can build on it.
What it costs
Training runs on two machines, short runs on the road, large ones at home. A full base run over 80,000 steps:
| Machine | 80,000 steps | Power |
|---|---|---|
| MacBook Air M416 GB | 40 h | 0,45 € |
| RTX 306012 GB | 11,4 h | 1,30 € |
| RTX 309024 GB | 4,5 h | 0,85 € |
At an electricity price of 0.37 euro per kilowatt hour.
Where the journey goes
The models are named by a fixed scheme: the size class comes from the parameter count, and the version number only rises once a previously named target is met on the bench. Star 3, for example, requires the net to retrieve world knowledge from its base in a meaningful way for the first time. No name can claim what the measurement does not support.
The destination is called Cosmos 1 and sits two size classes above where things stand today: a model that can stand next to the first public ChatGPT. By today's standards its wow effect was modest, but it was a model to be taken seriously. My machines will not carry it all the way there; in individual areas I think it is doable. The road is broken into stages, and every one of them is measured before it earns a name.
