What does the causal attention mask look like in the attention weights matrix?

In Vizuara's Lecture 7 the causal mask turns the square attention weights matrix into a lower-triangular one: every entry above the diagonal is 0. Because a context vector is an attention-weighted sum of value vectors, a zero weight means the matching future value vector contributes nothing, without the value vectors themselves being touched. Each row still sums to 1, so the first row becomes 1, 0, 0 and so on, the second might be 0.48 and 0.52, and the third 0.33, 0.37 and 0.3.

• For the first token, the context vector is simply 1 times v1; for the second it is 0.48 v1 plus 0.52 v2.

▸In their words

“So here this is a very smart mathematical trick. So we are not touching the value vectors.” Sreedath Panat · at 14:21 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-24). Read the whole episode summary →

▸More from this episode