Toolssmall, simple, playable
Tiny Tools
How a language model splits text into tokens.
- gpt-4o
- o200k_base
- runs locally
A tiny language model, trained on this site, picking words with weighted dice.
- trigram model
- top-k
- sampling
One neuron and one straight line, learning to split dots (and failing on XOR).
- classifier
- learning rate
- XOR
How surprising a text is, in bits per character, plus a guess-the-next-letter game.
- shannon
- information
- compression
How close a text is to aircraft-manual English: short sentences, simple words.
- simple words
- short sentences
- AI answers
A simulated Jev call: answers as a choice, a score or a yes/no, each with a probability.
- choice
- score
- noul
A small brass machine that lets requests through in a burst, then at a steady rate.
- rate limiting
- bursts
- APIs
- token counter
- temperature dial
- perceptron
- entropy meter
- simplified english
- typed decisions
- token bucket
- calibration game
- vector angles
Seriestransformers from scratch
Transformers from scratch
One small machine for each idea inside a transformer, in order. A and B take attention apart; C puts it back together with real words; D adds the mask that lets each word look only back.
Query, key and value as drawers: each drawer opens by how well its key matches the query.
- query
- key
- softmax
The output: each drawer gives weight × sand, and a cup adds the three parts.
- value
- weighted average
- output
cat, mat and it each ask with their own query, and each gets its own output.
- self-attention
- named numbers
- new meaning
The causal mask: a blind drops over every word after the one that asks, as when GPT writes.
- causal mask
- GPT
- one word at a time
Small experiments in how things work.
- server
- none
- sends
- nothing