How does attention fix the context vector bottleneck in machine translation?

The Vizuara instructor previews that attention lets the decoder look back at every source token and weight them differently at each step. Translating 'the cat sat today' into French, when the decoder writes 'chat' it might put 0.75 of its attention on 'cat' and only small weights such as 0.05 on 'sat' and 'today', and those weights change for each word it produces. A fixed context vector cannot shift focus this way.

• The instructor stresses that context vectors are not bad in themselves; the flaw is capturing them by passing tokens sequentially through an RNN.
• Attention scores themselves are the subject of the next lecture.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-04). Read the whole episode summary →

▸More from this episode