Why can't self-attention use raw input embeddings for attention scores?

Sreedath Panat of Vizuara argues that input embeddings ignore context: the same word always has the same vector wherever it appears. In the sentence where a dog chases a ball but it could not catch it, the first it refers to the dog and the second to the ball, yet both share one embedding, so a plain X times X-transpose score cannot treat them differently. Transforming embeddings with learned matrices before taking the dot product is what lets the scores reflect what each token should attend to.

• Lecture 5's simplified self-attention used softmax of X times X transpose over the square root of d, multiplied by X itself.
• Lecture 6 swaps each of those three X's for a query, a key and a value.

▸In their words

“So we can't really rely on just the input embeddings alone to construct the attention scores.” Sreedath Panat · at 10:12 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-18). Read the whole episode summary →

▸More from this episode