Why is dot product attention scaled by the square root of d_k, according to Vizuara?

The Vizuara instructor explains that the dot product of two randomly initialised query and key vectors grows in magnitude with the square root of their dimension. At the hundreds or thousands of dimensions real models use, raw scores get large enough that softmax saturates and hands almost all the weight to one token. Dividing every score by the square root of d cancels that growth, so the scores stay in a similar range whatever the dimension and the softmax weights stay meaningful.

• Going from 1 to 64 dimensions raises the typical score about 8-fold, because the square root of 64 is 8.
• With more dimensions the dot product sums more pairwise products, and many positive terms add up into a bigger raw score.

▸In their words

“This number is proportional to square root of dimensionality.” Instructor · at 11:58 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-11). Read the whole episode summary →

▸More from this episode