Why does self-attention use softmax instead of dividing attention scores by their sum?

In Vizuara's Lecture 5, Sreedath Panat notes that dividing scores such as 1, 2, 3 and 6 by their total of 12 does give numbers that sum to 1. The problem is that dot products turn negative when two vectors are more than 90 degrees apart, and dividing by the sum would then yield negative attention weights. Softmax guarantees every weight is positive and that each row sums to 1, which is why raw scores are always passed through it.

• Sreedath Panat says a weight near 0 should mean two tokens do not attend to each other and near 1 that they strongly do.
• An interactive Vizuara attention lab lets learners compare dividing by the sum with softmax.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode