How does an RNN encoder build the context vector in neural machine translation?

In Vizuara's explanation, the encoder is one recurrent neural network applied over several time steps. At each step it takes the previous hidden state and the embedding of the current token and outputs a new hidden state: h0, which starts as a random vector, plus the first token gives h1, and so on. For the four-word sentence 'this is a car', the final hidden state h4 is the context vector carrying the sentence's meaning.

• The diagram's repeated RNN boxes are time steps of the same network, not separate networks in sequence.
• For a longer sentence the context vector would be h20 or h50 instead of h4.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-04). Read the whole episode summary →

▸More from this episode