What is the difference between dot product attention and Bahdanau additive attention?

In Vizuara's Lecture 4, additive (Bahdanau) attention scores a pair of hidden states with a small neural network: each is multiplied by its own weight matrix, the results are added, passed through tanh and then dotted with a learned vector v. Luong's dot product attention, published in 2015 a year after Bahdanau's paper, drops all of that and simply takes the dot product of the two vectors. It has no learnable parameters at all, and the instructor argues a scoring step does not need to be more complex than that.

• The instructor cites roughly 11,000 to 12,500 citations for the Luong paper.
• In the lecture's notation, additive is v dot tanh(W1 s + W2 h), dot product is s dot h, and scaled dot product divides s dot h by the square root of d.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-11). Read the whole episode summary →

▸More from this episode