How does softmax saturation hurt LLM training, according to Vizuara?

The Vizuara instructor argues that a model in early training needs its predictions to be unconfident, because a less confident correct prediction produces a higher loss, larger gradients and therefore real weight updates. Unscaled scores break this: near-equal raw scores of 754, 728, 741 and 739 came out of softmax as about 99.999% for the first and close to zero for the rest. The model looks certain when it is not, so the loss shrinks and learning stalls. After scaling, the same scores gave about 31%, 19%, 24% and 24%.

• The loss example compares a prediction of 0.99, 0.005, 0.005 with 0.6, 0.2, 0.2 against a true label of 1, 0, 0; the second gives the higher loss.
• The instructor says the same argument applies to softmax over next-token probabilities, shown with candidate tokens such as cat, sat and down.

▸In their words

“The softmax will convert this low confidence prediction into a high confidence prediction.” Instructor · at 25:05 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-11). Read the whole episode summary →

▸More from this episode