The attention family cover art

Questions

Questions answered on The attention family

60 questions from 9 episodes of The attention family, each answered in our own words from our transcript of the episode.

▸Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9

2026-10-09 · 8 questions

Q1Why do transformers need position embeddings, according to Vizuara?Q2Does adding a position embedding change a word's meaning in a transformer?Q3How do learned absolute position embeddings work in GPT-2?Q4How do sinusoidal position embeddings work?Q5Why does every position get a unique sinusoidal embedding if sine waves repeat?Q6How does Shaw et al.'s relative position representation work?Q7What is T5's relative position bias and how is it different from a position embedding?Q8What happens to a small language model trained without position embeddings?

▸Introduction to Multi-Head Attention (MHA) | The attention family series - Lecture 8

2026-10-03 · 6 questions

Q1Why do transformers use multiple attention heads, according to Vizuara?Q2How are the outputs of multiple attention heads combined into one context vector per token?Q3How many dimensions does each attention head get in GPT-2 and GPT-3?Q4Why is implementing multi-head attention with a for loop slow?Q5What is the weight split approach to multi-head attention that Vizuara teaches?Q6Why do you transpose dimensions 1 and 2 when computing batched multi-head attention?

▸Causal attention (masked self attention) | The attention family series - Lecture 7

2026-09-24 · 6 questions

Q1Why does causal attention mask future tokens, according to Vizuara?Q2What does the causal attention mask look like in the attention weights matrix?Q3Why is zeroing attention weights after softmax the wrong way to do causal masking?Q4How does masking attention scores with minus infinity before softmax work?Q5How is dropout applied to attention weights, and why scale by 1/(1-p)?Q6What are the full steps of causal self-attention in Vizuara's attention family series?

▸Q, K, V intuition - self attention with trainable weights | The attention family series - Lecture 6

2026-09-18 · 6 questions

Q1What do query, key and value mean in self-attention, according to Vizuara?Q2Why can't self-attention use raw input embeddings for attention scores?Q3How does a key weight matrix change attention scores in Vizuara's dog and ball example?Q4What does an attention-weighted sum of value vectors mean in the transformer formula?Q5Can the query, key and value space have a different dimension from the input embeddings?Q6How are the W_Q, W_K and W_V weight matrices learned in an LLM?

▸Simplified self attention | The attention family series - Lecture 5

2026-09-16 · 7 questions

Q1How does simplified self-attention compute context vectors without trainable weights, according to Vizuara?Q2What is the matrix formula for simplified self-attention, and how does it relate to Attention Is All You Need?Q3What is the difference between self-attention and Bahdanau attention?Q4Why does a word need a context vector instead of its embedding, according to Vizuara?Q5Why doesn't a token always get its highest attention score against itself?Q6Why does self-attention use softmax instead of dividing attention scores by their sum?Q7Are self-attention scores and attention weights symmetric?

▸Dot product attention and scaled dot product attention | The attention family series - Lecture 4

2026-09-11 · 6 questions

Q1Why is dot product attention scaled by the square root of d_k, according to Vizuara?Q2What is the difference between dot product attention and Bahdanau additive attention?Q3Why does a dot product work as an attention score between two tokens?Q4What happens to softmax attention weights without scaling at 1,000 or 4,096 dimensions?Q5How does softmax saturation hurt LLM training, according to Vizuara?Q6Which paper introduced scaled dot product attention and why is it used everywhere?

▸Additive Attention (Bahdanau Attention) | The Attention Family, Lecture 3

2026-09-08 · 7 questions

Q1How does Bahdanau additive attention compute alignment scores, according to Vizuara?Q2Why is Bahdanau attention called additive attention?Q3What problem did Bahdanau attention solve in RNN sequence-to-sequence translation?Q4What is the difference between attention scores and attention weights in Bahdanau attention?Q5How do you read an attention weights matrix in neural machine translation, according to Vizuara?Q6How does a Bahdanau attention decoder predict the next token, as Vizuara explains it?Q7How does additive attention differ from Luong dot-product and scaled dot-product attention?

▸Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2

2026-09-04 · 7 questions

Q1Why is a single context vector a bottleneck in seq2seq, according to Vizuara?Q2How does an RNN encoder build the context vector in neural machine translation?Q3What does the decoder rely on in a pre-attention seq2seq translation model?Q4Why do RNN seq2seq models struggle with long-range dependency, according to Vizuara?Q5At what sentence length does seq2seq translation quality drop without attention?Q6How does attention fix the context vector bottleneck in machine translation?Q7When were Bahdanau, dot-product and scaled dot-product attention introduced?

▸The Attention Family: Every Attention Mechanism, Explained | New Series Launch | Lecture 1

2026-09-01 · 7 questions

Q1What are the seven buckets of attention mechanisms in Vizuara's attention family series?Q2How many types of attention mechanisms are there, according to the Vizuara attention lectures?Q3How does Vizuara's attention family series compare attention in Llama 3, DeepSeek V3 and Kimi Linear?Q4Why is attention quadratic in the number of tokens, according to the Vizuara attention lectures?Q5Is flash attention a new attention architecture or an implementation trick, per Vizuara's lectures?Q6What does Vizuara's attention family lecture series cover, and in what order?Q7Do you need to know transformers before starting Vizuara's attention family lectures?

Questions from every show →

Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →