How does Bahdanau additive attention compute alignment scores, according to Vizuara?

The Vizuara instructor explains that each score comes from a tiny neural network that takes the previous decoder state s and one encoder hidden state h. It multiplies s by a weight matrix W1 and h by W2, adds the two projected vectors, applies tanh for non-linearity, then takes a dot product with a learned vector v to collapse the result into a single number: score = v · tanh(W1·s + W2·h).

• The same scoring network is applied to every encoder state, so a four-word sentence yields four scores per decoding step.
• W1 and W2 project both states into a shared space so they can be added element by element.
• The vector v is not hand-designed; it is learned during training along with W1 and W2.

▸In their words

“Vector v is actually a learned parameter; it is not something which we define.” Instructor · at 33:58 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-08). Read the whole episode summary →

▸More from this episode