podcasts / The attention family / episode / question
Question

Why do RNN seq2seq models struggle with long-range dependency, according to Vizuara?

Answered from Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2 · The attention family · 2026-09-04

Because an RNN reads tokens in order, information from the start of a long sentence tends to fade by the time the final hidden state is formed, which keeps more of the sentence's end. In Vizuara's example, 'the teacher who was teaching a difficult concept to the students smiled', the word smiled sits right next to students but belongs to teacher at the very start, a link a context-vector model finds hard to keep.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-04). Read the whole episode summary →

▸More from this episode

Q1Why is a single context vector a bottleneck in seq2seq, according to Vizuara?Q2How does an RNN encoder build the context vector in neural machine translation?Q3What does the decoder rely on in a pre-attention seq2seq translation model?Q5At what sentence length does seq2seq translation quality drop without attention?Q6How does attention fix the context vector bottleneck in machine translation?Q7When were Bahdanau, dot-product and scaled dot-product attention introduced?
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →