How does simplified self-attention compute context vectors without trainable weights, according to Vizuara?

In Vizuara's Lecture 5, Sreedath Panat computes a context vector for one token in three steps. First he takes the dot product of that token's embedding with every embedding in the sentence, itself included, to get raw scores. He divides the scores by the square root of the embedding dimension and applies softmax so they become positive weights that sum to 1. Finally he multiplies each embedding by its weight and adds them up. No learned matrices are involved; the input embeddings do all the work.

• Raw scores are written omega, the softmax weights alpha and the resulting context vector z.
• For the token big in the example sentence, the resulting context vector was 0.5182, 0.3431, 0.5794.
• The input embeddings themselves are assumed to come from an existing embedding table such as Word2Vec.

▸In their words

“It's simple dot product.” Sreedath Panat · at 17:58 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode