What does the decoder rely on in a pre-attention seq2seq translation model?

The Vizuara instructor says the decoder depends only on the context vector. It is another recurrent neural network that takes the context vector as its first input, outputs the first French token and a hidden state s1, then feeds s1 and the embedding of the token it just produced into the next step, until an end-of-sequence token. It has no direct view of the original English sentence, which is the core weakness of the design.

• Each decoder step produces two outputs: an output vector for the next French word and a new hidden state.
• Source and target sentences need not have the same length or word order, as in translating 'European economic zone' into French.

▸In their words

“Decoder knows nothing else. Decoder has no idea what sentence was passed into the encoder.” Instructor · at 16:45 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-04). Read the whole episode summary →

▸More from this episode