Toolstransformers from scratch · D of 4
Only Look Back
A transformer that writes text, such as GPT, writes one word at a time, so a word cannot use the words that come after it. The causal mask enforces this. The cabinet is the one from figure C, with the words in sentence order; a blind now drops over the drawer of every word after the word that asks.
~/tools/transformers/look-back
cat asks · weights 1.0 — — · output (1.0, .00)
What is what
- Drawers
- cat is blue, mat is orange, it is green, in sentence order. A drawer opens by its weight.
- Blinds
- The mask. A blind covers each word that comes after the word that asks; that drawer stays shut, with weight 0.
- Cups
- One on top for each word: its output with the mask.
- Readout
- The weights of the word that asks; a dash marks a masked word.
How to play
- A click on the drawing, or the Next word button, passes the question to the next word. The keys 1, 2 and 3 pick cat, mat and it.
- The Mask button, or the M key, turns the mask off and on, to compare with figure C.
- cat asks: blinds cover mat and it, so cat uses only itself, and its cup holds its own value, (1, 0). Without the mask it was (.42, .42).
- mat asks: a blind covers it, so cat and mat share the weight half and half, (.5, .5).
- it asks: it is the last word, so no blind drops, and the result is the same as in figure C.
Without the mask, attention sees all the words at once and does not know their order; transformers add each word's position to its numbers, which is called positional encoding. The numbers here are examples, and this figure does not divide q·k by √2.