Are self-attention scores and attention weights symmetric?

Sreedath Panat shows that in simplified self-attention the raw scores are symmetric: omega 1 2 equals omega 2 1, because the dot product of x1 and x2 is the same in either order, and both are divided by the same square root of d. The attention weights are not guaranteed to be symmetric. Each weight comes from a row-wise softmax that depends on all the other scores in that row, so alpha 1 2 generally differs from alpha 2 1.

• For a six-token sentence the weight matrix holds 36 attention weights.
• Each row sums to 1, but the columns need not.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode