Answered from Dot product attention and scaled dot product attention | The attention family series - Lecture 4 · The attention family · 2026-09-11
The Vizuara instructor's reasoning is geometric. Tokens are vectors in one shared high-dimensional semantic space, and related words, such as cat, dog and tiger, or apple, orange and pineapple, point in similar directions. The dot product tracks the angle between two vectors: it is a high positive number when they point the same way, exactly zero at 90 degrees and a large negative number at 180 degrees. Tokens that are semantically related therefore get high scores, so the dot product can be used directly as the attention score.
Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-11). Read the whole episode summary →