What is the difference between self-attention and Bahdanau attention?

According to Sreedath Panat of Vizuara, Bahdanau attention lets the tokens of an output sequence, such as a French translation, attend to the tokens of a different input sequence, such as the English source. The two sequences can have different lengths, so the attention map is generally rectangular, which makes it closer to cross-attention. In self-attention a single sentence attends to itself, so every token scores against every token of the same sequence and the attention weight matrix is always square.

• Bahdanau's additive attention also used learned weights and a tanh to compute scores, rather than a plain dot product.
• The lecture puts the Bahdanau paper at almost 45,000 citations.
• Cross-attention is covered in a later lecture of the series.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-16). Read the whole episode summary →

▸More from this episode