What is the matrix formula for simplified self-attention, and how does it relate to Attention Is All You Need?

Sreedath Panat writes simplified self-attention as Z equals softmax of X times X transpose, divided by the square root of d, multiplied by X, where X holds one embedding per row and d is the embedding dimension. The Attention Is All You Need formula has the same shape, softmax of Q times K transpose over the square root of d, multiplied by V. The simplified version is what that formula becomes when query, key and value are all set equal to the input X.

• In the lecture's example, X is 6 by 3, X times X transpose is 6 by 6, and Z is 6 by 3.
• Softmax is taken row-wise, so each row of the 6 by 6 weight matrix sums to 1.
• With three-dimensional toy vectors, the scaling factor is the square root of 3.

▸In their words

“In today's lecture we are having query equal to key equal to value equal to x” Sreedath Panat · at 36:06 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode