Why do transformers use multiple attention heads, according to Vizuara?

In Vizuara's Lecture 8, Sreedath Panat explains that a single attention head produces one set of attention weights and therefore only one context vector per token, which cannot capture everything worth knowing about that token. With several heads, each computes its own independent attention weights, so a token gets one context vector per head, and each head can learn something different. Combined, those outputs give a richer representation than one head can. Panat stresses this picture is qualitative: what a given head really learns is hard to say without attention visualisation maps.

• The example sentence is 'The dog chased the ball but it could not catch it', where one 'it' refers to the ball and the dog is the one acting.
• Suggested head roles, such as noun versus not, or animate versus inanimate, are offered as deliberately oversimplified illustrations.

▸In their words

“It's very difficult to unless you have some sort of attention visualization map.” Sreedath Panat · at 7:56 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-03). Read the whole episode summary →

▸More from this episode