What happens to softmax attention weights without scaling at 1,000 or 4,096 dimensions?

In the Vizuara instructor's Colab notebook, a query scored against two random keys at 1,000 dimensions produced raw scores of 252 and 258. Softmax on those gave about 0.2% and 99.7%, while dividing by the square root of 1,000 (about 31) first gave roughly 45% and 54%. At 4,096 dimensions, raw scores of 1,032 and 1,042 became about 0.003% and 99.996% unscaled, but 16.13 and 16.29 after dividing by 64, which softmax turned into roughly 46% and 54%.

• At 1 dimension scaling changes nothing: both versions gave weights of 0.544 and 0.455.
• At 4 dimensions the effect is mild: unscaled weights of 0.62 and 0.30 against scaled 0.56 and 0.43.
• Softmax exponentiates the scores, so e to the 252 is negligible next to e to the 258.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-11). Read the whole episode summary →

▸More from this episode