What does an attention-weighted sum of value vectors mean in the transformer formula?

Vizuara's Sreedath Panat unpacks the Attention Is All You Need formula, softmax(QK transpose over the square root of d) times V, as a weighted sum: each token's context vector is the sum over j of attention weight w_j times value vector v_j. For the word ball in a five-token sentence, softmax weights of 0.17, 0.16, 0.25, 0.10 and 0.31 multiply value vectors v1 to v5, and their sum is the context vector z5. Changing the query to token 3 gives a new set of weights and z3.

• The context vector lives in the value space, so in the lecture's example it is two-dimensional even though the input embedding was three-dimensional.
• Panat says the formula can look intimidating but is simple once read as this weighted sum.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-18). Read the whole episode summary →

▸More from this episode