The attention family

The Attention Family: Every Attention Mechanism, Explained | New Series Launch | Lecture 1

📚 Part 1 of 9 ⏱ 28 min listen · ~2 min read

It gives you one seven-slot framework for placing any attention variant, so the next model paper reads as a change to a known part rather than new jargon.

Key takeaways

01Vizuara's Attention Family series counts more than 50 reasonably famous attention variants and sorts them instead of teaching them as a list.
02Vizuara's seven buckets are scoring, connectivity, heads and KV cache, sub-quadratic cost, GPU implementation, position information and domain.
03Standard attention costs n squared because each of n queries takes a dot product with each of n keys; sparse and linear designs push the exponent below 2.
04Llama 3, DeepSeek V3 and Kimi Linear share buckets 1, 2 and 7 but split on heads, position encoding and sub-quadratic tricks such as Kimi Delta attention.

Chapters

0:00 · Why attention now needs a family tree2:04 · The instructor and who the series is for3:08 · Three models, three different attention recipes4:37 · A wall of 35-odd attention names6:11 · Grouping mechanisms instead of memorising them7:01 · The seven buckets, from scoring to domain+7 more

Read the full summary →

Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2

📚 Part 2 of 9 ⏱ 26 min listen · ~2 min read

It gives a clear mental picture of the context vector and exactly why a single one breaks, which is the setup every later attention lecture builds on.

Key takeaways

01Vizuara's Lecture 2 covers neural machine translation, the RNN encoder-decoder that translated English into French before attention existed.
02An RNN encoder folds each token into a hidden state, and the last state, h4 for a four-word sentence, becomes the one context vector the decoder ever sees.
03An 800-dimension context vector leaves about 100 dimensions per word for 8 words but only 20 per word for 40 words, in the instructor's rough analogy.
04RNN encoders lose early tokens: in 'the teacher who was teaching the students smiled', linking smiled back to teacher is hard for seq2seq models.

Chapters

0:00 · Why study what existed before attention1:35 · Recap of the seven-bucket map of attention variants6:01 · English-to-French example with uneven word order7:11 · Tokens, embeddings and cosine similarity9:44 · Encoder, decoder and the single context vector11:28 · Encoder RNN time steps building hidden states+8 more

Read the full summary →

Additive Attention (Bahdanau Attention) | The Attention Family, Lecture 3

📚 Part 3 of 9 ⏱ 36 min listen · ~2 min read

It derives the additive score formula step by step from a worked English-to-French example, which makes the later jump to dot-product and self-attention easy to follow.

Key takeaways

01Bahdanau attention, published in 2014, targeted RNN translators that degraded once source sentences passed about 20 words because of one fixed context vector.
02Bahdanau's decoder builds a fresh context vector for every output token as a weighted sum of all encoder hidden states, weighted by relevance to that step.
03Vizuara's example gives cat a weight of 0.75 while the decoder writes chat, with the other three English words sharing the remaining 0.25.
04Additive scoring needs a small neural network per encoder-decoder pair: v dot tanh(W1·s + W2·h), with W1, W2 and v all learned during training.

Chapters

0:00 · Recap of the fixed context vector bottleneck2:06 · Seven buckets for classifying attention mechanisms5:18 · A context vector that changes with every output token8:54 · Context vectors as weighted sums of encoder states11:54 · Attention scores versus attention weights13:02 · Simplified encoder-decoder diagram with attention+8 more

Read the full summary →

Dot product attention and scaled dot product attention | The attention family series - Lecture 4

📚 Part 4 of 9 ⏱ 30 min listen · ~2 min read

Thirty minutes of worked numbers that make the square root of d_k in the transformer attention formula obvious rather than memorised.

Key takeaways

01Vizuara's Lecture 4 compares three attention scorers: Bahdanau's additive network, Luong's 2015 dot product and the scaled version from Attention Is All You Need.
02Dot product attention has no learnable parameters: two vectors pointing the same way score high, a 90-degree angle scores 0 and opposite directions score negative.
03Raw dot products of random vectors grow with the square root of dimension, so moving from 1 to 64 dimensions makes typical scores about 8 times larger.
04At 4,096 dimensions, raw scores of 1,032 and 1,042 gave softmax weights under 0.01% and over 99.99%; dividing by 64 first gave about 46% and 54%.

Chapters

0:00 · Recap of Bahdanau additive attention2:01 · Where scoring sits in the series map2:46 · The Luong 2015 paper and Attention Is All You Need3:43 · Three-part plan for the lecture4:45 · Swapping the scoring network for a dot product6:40 · Cosine similarity in a semantic space+8 more

Read the full summary →

Simplified self attention | The attention family series - Lecture 5

📚 Part 5 of 9 ⏱ 46 min listen · ~2 min read

A slow, numeric walk from one token's dot products to the full matrix formula, which makes the Attention Is All You Need equation readable before query, key and value arrive.

Key takeaways

01Vizuara's Lecture 5 builds self-attention with no trainable weights: query, key and value are all the raw input embedding X of the sentence.
02Self-attention lets every token of one sentence attend to the same sentence, so the weight matrix is square, unlike Bahdanau's rectangular cross-language map.
03For the six-token sentence dream big and work for it, X is 6 by 3, X times X transpose gives 36 scores, and the context vectors come out 6 by 3.
04Dot products reward vector length, so in the Colab example big scores 1.04 against work but only 0.91 against itself.

Chapters

0:00 · Recap of Bahdanau and dot product scoring3:10 · Moving from scoring to the connectivity bucket4:54 · Roadmap through masked and multi-head attention8:21 · Cross-attention versus self-attention matrix shapes11:00 · Two cat sentences where 'it' changes meaning13:59 · Six toy vectors for a six-word sentence+8 more

Read the full summary →

Q, K, V intuition - self attention with trainable weights | The attention family series - Lecture 6

📚 Part 6 of 9 ⏱ 49 min listen · ~2 min read

A slow, numbers-first walk through the Q, K, V formula that turns the transformer's most quoted equation into arithmetic you can check by hand.

Key takeaways

01Vizuara's Lecture 6 replaces the X-times-X-transpose scores of simplified self-attention with learned query, key and value weight matrices.
02Raw embeddings fail on the sentence where it means dog once and ball once: both copies of it share one vector, so their dot products cannot differ.
03Sreedath Panat's toy example scores it at 0.51 against both dog and ball; projecting only the keys lifts ball to 0.96 and dog to 0.56.
04W_Q, W_K and W_V are not known in advance: Panat says they start as random matrices and learning them is a big part of training an LLM.

Chapters

0:00 · Recap of simplified self-attention from Lecture 52:28 · Where trainable self-attention sits in the series5:11 · Three-part plan built around the Q, K, V formula7:42 · Why raw dot products confuse two copies of it11:12 · Equal scores for dog and ball in three dimensions12:58 · A key weight matrix that separates dog from ball+7 more

Read the full summary →

Causal attention (masked self attention) | The attention family series - Lecture 7

📚 Part 7 of 9 ⏱ 51 min listen · ~2 min read

A clear account of why the causal mask goes in before the softmax rather than after it, with worked rows and code.

Key takeaways

01Vizuara's Lecture 7 explains causal attention: a text model in training must only use tokens it would actually have at inference time.
02Causal attention zeroes the upper triangle of the attention weights matrix, so future value vectors get weight 0 while each row still sums to 1.
03Masking a row of scores 2, 3 and 5 with minus infinity in the last two slots gives softmax weights of 1, 0 and 0, since e to the minus infinity is 0.
04Zeroing weights after softmax and renormalising still leaks the future, Sreedath Panat warns, because the softmax denominator already used future keys.

Chapters

0:00 · Recap of query, key and value projections2:36 · Causal masking on the series connectivity map4:47 · Three-part plan: hiding, masking methods, dropout6:06 · Training versus inference on Dream big and work for it10:14 · Upper triangle of zeros in the weights matrix16:34 · Zeroing weights after softmax and renormalising rows+8 more

Read the full summary →

Introduction to Multi-Head Attention (MHA) | The attention family series - Lecture 8

📚 Part 8 of 9 ⏱ 71 min listen · ~2 min read

A shape-by-shape walk through multi-head attention that makes the view, transpose and output projection lines in transformer code make sense.

Key takeaways

01Vizuara's Lecture 8 adds multiple attention heads to the masked, scaled self-attention built in Lecture 7 of The attention family series.
02Sreedath Panat's two-head example splits a 6-dimensional embedding into two 3-dimensional heads, concatenates them and mixes them with a 6 by 6 output matrix.
03GPT-2 used 12 heads over a 768-dimensional embedding for 64 dimensions per head; GPT-3 used 96 heads over 12,288 for about 128.
04Looping over heads one at a time took 0.003 seconds for 2 heads but 0.63 seconds for 4,000 in Panat's Colab timing of one forward pass.

Chapters

0:00 · Recap of causal masking and dropout6:11 · Why one attention head cannot capture everything10:18 · Two heads sharing a six-dimensional embedding12:36 · Concatenating head outputs along the embedding axis15:55 · Output projection that mixes the heads together18:44 · Head size as d over n in GPT-2 and GPT-3+8 more

Read the full summary →

Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9

📚 Part 9 of 9 ⏱ 73 min listen · ~2 min read

A clear map of where position information enters attention, with the arithmetic behind each method worked through by hand.

Key takeaways

01Vizuara's Lecture 9 shows why 'all dogs are animals' and 'all animals are dogs' look identical to attention unless position information is added.
02Sreedath Panat sorts position methods by where they plug in: GPT-2 and sinusoidal add to the input, Shaw adds to keys, T5 adds to the query-key score.
03GPT-2's learned position table has 1,024 rows of 768 numbers, one per slot in its context window, beside a token table of about 50,000 rows.
04GPT-2's learned table has no vector for position 1,100, the gap the 2017 sine and cosine formula closes by producing a unique vector for any position.

Chapters

0:00 · Recap of multi-head attention and the plan4:25 · Same words in two orders, same attention scores9:50 · Why added position does not turn dogs into cats18:14 · Places where position can enter attention22:02 · GPT-2 and BERT's learned position table26:04 · Context window limit of a learned table+8 more

Read the full summary →

The Attention Family: Every Attention Mechanism, Explained | New Series Launch | Lecture 1

Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2

Additive Attention (Bahdanau Attention) | The Attention Family, Lecture 3

Dot product attention and scaled dot product attention | The attention family series - Lecture 4

Simplified self attention | The attention family series - Lecture 5

Q, K, V intuition - self attention with trainable weights | The attention family series - Lecture 6

Causal attention (masked self attention) | The attention family series - Lecture 7

Introduction to Multi-Head Attention (MHA) | The attention family series - Lecture 8

Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9

▸Questions this show answers