podcasts / The attention family / episode / question
Question

Why doesn't a token always get its highest attention score against itself?

Answered from Simplified self attention | The attention family series - Lecture 5 · The attention family · 2026-09-16

Sreedath Panat explains that embedding vectors do not all have length 1, and a dot product rewards magnitude as well as direction. A longer vector pointing in a similar direction can therefore outscore the token's own vector. In the lecture's example the token big scored 0.91 against itself but 1.04 against work, which has a larger magnitude, and work stayed the dominant weight after softmax at about 0.206.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode

Q1How does simplified self-attention compute context vectors without trainable weights, according to Vizuara?Q2What is the matrix formula for simplified self-attention, and how does it relate to Attention Is All You Need?Q3What is the difference between self-attention and Bahdanau attention?Q4Why does a word need a context vector instead of its embedding, according to Vizuara?Q6Why does self-attention use softmax instead of dividing attention scores by their sum?Q7Are self-attention scores and attention weights symmetric?
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →