Why does a word need a context vector instead of its embedding, according to Vizuara?

Sreedath Panat uses two sentences: the cat drank the milk because it was thirsty, and the cat drank the milk because it was sweet. The token it has the same base embedding in both, yet it means the cat in the first and the milk in the second. A fixed embedding cannot capture that, so self-attention modifies each base embedding using the rest of the sentence, producing a context vector that differs whenever the surrounding words differ.

• In the Colab plot, the context vector for big points in a slightly different direction from its original embedding.
• The lecture calls converting embedding x into context vector z the whole job of self-attention.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode