What is the difference between attention scores and attention weights in Bahdanau attention?

In Vizuara's lecture, attention scores are the raw numbers produced by the scoring network for each encoder state, and they need not sum to one. Passing them through softmax (e to the x_i divided by the sum of e to the x_j) turns them into attention weights that form a probability distribution, and those weights multiply the encoder hidden states to build the context vector.

• Example raw scores of 2, 4, 1 and 3 keep their ranking after softmax.
• Example weights of 0.05, 0.75, 0.15 and 0.05 sum to one and give the context vector C2.
• The instructor notes that later normalisation tricks for softmax were not part of the Bahdanau paper.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-08). Read the whole episode summary →

▸More from this episode