Toolstransformers from scratch · A of 4
Query Meets Key
The first half of attention, as a cabinet with three drawers. Each drawer is one token: the arrow on its front is its key, and the sand inside is its value. The dial on the right holds the query. Each drawer opens by its weight, which says how well its key matches the query.
~/tools/transformers/search-cabinet
weights .80 .14 .06
What is what
- Drawers
- One drawer for each token: blue, orange and green. A drawer opens by its weight.
- Key arrows
- On the drawer fronts. What each token is.
- Query arrow
- Purple, on the big dial. What the search looks for.
- Sand
- The value of each token. Figure B puts it to use.
- Scales
- Under each drawer: its weight, from 0 to 1. The readout shows the same three weights.
How to play
- Moving the pointer left and right across the drawing turns the query.
- The arrow keys turn it 10° at a time (45° with Shift). The 0 key points it straight up.
- With no input, the dial turns by itself and visits each drawer.
- When the query points the same way as a key, that drawer opens most.
- Between two keys, both drawers open about half. Straight up is a good place to check this.
- The three openings always add up to one full opening: when one opens more, the others close a little.
The match is the dot product q·k / √2: high when the two arrows point the same way, low when they point apart. Softmax turns the three matches into three weights that add up to 1. The directions of the arrows and the amounts of sand are examples.