How many dimensions does each attention head get in GPT-2 and GPT-3?

Vizuara's Lecture 8 gives the usual rule: with an embedding dimension d and n heads, each head works in d divided by n dimensions. GPT-2 had a 768-dimensional embedding and 12 heads, so each head works in 64 dimensions. GPT-3 had a 12,288-dimensional embedding and 96 heads, which Sreedath Panat rounds to about 128 dimensions per head. Panat notes each head could in principle use the full embedding size, but compute would blow up, so the d over n split is what is done in practice.

• A toy example with 6 tokens, a 12-dimensional embedding and 4 heads gives 3 dimensions per head and 6 by 3 context vectors per head.
• Concatenating four 6 by 3 outputs restores the original 6 by 12 shape.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-03). Read the whole episode summary →

▸More from this episode