Nestor G Pestelos Jr · ELI5

Transformers

What Is Attention?

Attention decides which earlier words to mix into the word being read now.

01 / Pronoun

Which word is "it"?

The model scores earlier words and mixes their meaning into "it".

Query on it, with stronger weight on dog than llama Illustrative sentence: The dog chased the llama because it was hungry. Query it sends a thick mix to dog at 0.71 and a thin mix to llama at 0.19. The dog chased the llama because it was hungry. Illustrative example. The weights are made up. 0.71 0.19 it Query dog strong mix llama weak mix "it" keeps more of dog than of llama
How the mix answers the pronoun

A person might pick "dog" because hungry animals chase. The model compares "it" with earlier words and keeps a weighted blend of their meanings.

The 0.71 and 0.19 numbers are illustrative. A trained model would emit its own scores.

02 / Query, Key, Value

Score, then mix.

Query is the look-up, Key is the label, and Value is the payload that gets mixed in.

Query, Key, Value, then score, scale, softmax, and mix Three cards: Query as look-up, Key as label, Value as payload. Then softmax of Q K transpose over square root of d sub k, times V, shown as score, scale, softmax, mix. Query look-up Key label Value payload softmax( QKᵀ / √dₖ ) V weights from scores, then those weights mix the payloads 1. Score Q matches K 2. Scale divide by √dₖ 3. Softmax weights sum to 1 4. Mix weights × V
Why the scores get divided by √dₖ
  • Each word makes three vectors: Query, Key, and Value.
  • The score is how well this Query matches each Key.
  • Divide by √dₖ so the scores stay in a range softmax can use. Large raw dots push softmax toward 0 and 1 (Vaswani et al., §3.2.1).
  • Softmax turns the scaled scores into weights that add to 1.
  • Those weights mix the Values. The result is the new vector for this word.
03 / Masks

Who can look where.

Encoder words all see each other, and decoder words skip any word that has not been written yet.

Encoder full grid versus decoder causal lower triangle Two 4 by 4 grids. Encoder cells are all filled. Decoder cells above the diagonal are muted so a word cannot see later words. Encoder every word sees every word Decoder later words stay hidden I saw a cat I saw a cat I saw a cat I saw a cat can look blocked (muted)
Why a decoder hides the future
  • Rows are the word that is looking. Columns are the word being looked at.
  • An encoder may let every token attend to every token, including later ones in the same pass.
  • A decoder that writes one token at a time uses a causal mask. Future tokens do not exist yet, so those cells stay muted.