How are the outputs of multiple attention heads combined into one context vector per token?

Sreedath Panat of Vizuara shows that each head's context vectors are concatenated along the embedding dimension, so two heads of six three-dimensional vectors become one six by six matrix. Because concatenation leaves the first three dimensions owned by head one and the last three by head two, the result then goes through an output projection, a square matrix sized by the embedding dimension, which mixes the values so no dimension can be traced to a single head. The projection keeps the shape unchanged: 12 tokens in 6 dimensions stay 12 by 6.

• Panat argues the heads run in parallel and no head should get precedence just because it was labelled first.
• In the lecture's PyTorch code this mixing matrix is the out_proj layer of size d_out by d_out.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-03). Read the whole episode summary →

▸More from this episode