Why is a single context vector a bottleneck in seq2seq, according to Vizuara?

The Vizuara instructor explains that a pre-attention seq2seq model compresses the whole input sentence into one context vector of fixed size, for example 768 dimensions, and the decoder sees nothing else. A four-word sentence and a 40-word sentence get the same space, so the longer the input, the more meaning is crammed into the same vector and the more translation quality suffers.

The context vector lives in the same embedding space as the encoder's hidden states, so its dimensionality does not grow with sentence length.
The instructor calls this too much pressure on a single vector for long inputs.

▸In their words

“Too much pressure on one single vector. How can you expect one single vector to carry that much meaning?” Instructor · at 10:45 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-04). Read the whole episode summary →

▸More from this episode

Q2How does an RNN encoder build the context vector in neural machine translation?Q3What does the decoder rely on in a pre-attention seq2seq translation model?Q4Why do RNN seq2seq models struggle with long-range dependency, according to Vizuara?Q5At what sentence length does seq2seq translation quality drop without attention?Q6How does attention fix the context vector bottleneck in machine translation?Q7When were Bahdanau, dot-product and scaled dot-product attention introduced?

Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →