How is dropout applied to attention weights, and why scale by 1/(1-p)?

In Lecture 7, Sreedath Panat applies dropout to the causal attention weights during training only, so the weights cannot simply memorise the training text. Each element is dropped with probability p, and the survivors are multiplied by 1/(1-p) so that the average output magnitude stays the same. With p of 0.5 surviving weights double, for example 0.52 becomes 1.04, and with p of one third they are multiplied by 3/2.

• A weight of 0.27 kept half the time at 0.54 still averages 0.27 overall.
• Exactly 50% of elements are not guaranteed to drop; each one has a 50% chance.
• Dropout is switched off at inference, where all weights are used.

▸In their words

“The dropout is only done during training during inference you don't do dropout.” Sreedath Panat · at 34:37 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-24). Read the whole episode summary →

▸More from this episode