Answered from Q, K, V intuition - self attention with trainable weights | The attention family series - Lecture 6 · The attention family · 2026-09-18
Sreedath Panat gives it, dog and ball toy three-dimensional embeddings of (0.5, 0.5, 0.1), (0.9, 0.1, 0.1) and (0.1, 0.9, 0.1). Raw dot products give it 0.51 against both nouns. He then passes only the keys through a diagonal W_K matrix with entries 1, 2 and 1: dog becomes (0.9, 0.2, 0.1) and ball becomes (0.1, 1.8, 0.1). Now it scores 0.56 against dog and 0.96 against ball, so it attends more to ball, as the sentence requires.
Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-18). Read the whole episode summary →