How does additive attention differ from Luong dot-product and scaled dot-product attention?

The Vizuara instructor says additive (Bahdanau) attention computes each score with a small neural network, while Luong's 2015 paper showed the score can come from a plain dot product with no network at all. Scaled dot-product attention, introduced in Attention Is All You Need, became the scoring method used by nearly every later attention mechanism, and both are the subject of Lecture 4.

• The instructor recalls the Luong paper as having around 11,000 to 12,000 citations.
• All three methods sit in the scoring bucket of the series' seven-bucket taxonomy.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-08). Read the whole episode summary →

▸More from this episode