Why does causal attention mask future tokens, according to Vizuara?

Sreedath Panat of Vizuara explains that a language model generates text one token at a time, so at inference it never has value vectors, or keys, for tokens it has not produced yet. During training the whole sentence is available, but if the model learned to build context vectors from later tokens it would rely on information that disappears at inference. Causal attention makes training mimic inference: each position may only use the tokens up to and including itself.

• The lecture's example is predicting the token work from Dream big and, inside the six-token training sentence Dream big and work for it.
• The mask is applied during both training and inference, so generation behaves the way the model was trained.

▸In their words

“So if that is the behavior that you expect during inference, during training also you should inject that behavior.” Sreedath Panat · at 9:05 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-24). Read the whole episode summary →

▸More from this episode