Toolstransformers from scratch · C of 4
Every Word Asks
Attention with real words, from the sentence “The cat sat on the mat because it was tired.” Three words go in the cabinet: cat, mat and it. Each number now answers a plain question, and each word asks with its own query.
~/tools/transformers/every-word-asks
it asks · weights .67 .24 .09 · output (.67, .24)
What is what
- Drawers
- One word each: cat is blue, mat is orange, it is green.
- Keys and queries
- Two numbers: is it a thing? is it alive? Drawn as an arrow on a small grid: right is thing, up is alive.
- Values
- Two numbers: is it an animal? is it a thing to sit on? Held as sand in two compartments, animal on the left.
- Cups
- One on top for each word: the output of that word, in the same two columns.
How to play
- A click on the drawing passes the question to the next word. The keys 1, 2 and 3 pick cat, mat and it.
- With no input, the words take turns by themselves.
- it asks (1, 1): only cat answers yes to both questions, so the cat drawer opens most, and the it cup fills mostly with animal.
- cat asks (1, 0): cat and mat are both things, so they tie, and the cat cup is half animal, half thing to sit on.
- mat asks (0, 0): every match is 0, so all three weights are equal, and its output is the plain average.
| word | key (thing, alive) | query (thing, alive) | value (animal, sit on) |
|---|---|---|---|
| cat | (1, 1) | (1, 0) | (1, 0) |
| mat | (1, 0) | (0, 0) | (0, 1) |
| it | (0, 0) | (1, 1) | (0, 0) |
To keep the numbers simple, this figure does not divide q·k by √2. In a real model, all the words ask at the same time, and the model learns its own questions: hundreds of numbers, most with no simple name.