Answered from Introduction to Multi-Head Attention (MHA) | The attention family series - Lecture 8 · The attention family · 2026-10-03
In Vizuara's Lecture 8, weight split means joining the separate per-head query weight matrices, for example two 6 by 3 matrices, into one 6 by 6 query matrix, and doing the same for keys and values. One matrix multiplication then produces all queries at once, and the result is split into heads afterwards. Sreedath Panat prints both versions in Colab to show the first three columns of the big query matrix exactly match head one's queries and the last three match head two's, so the output is identical while the GPU does one large multiplication instead of many.
“So, what it shows is it does not matter if you split into two different matrix multiplication” Sreedath Panat · at 54:06 — open in the transcript →
Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-03). Read the whole episode summary →