Answered from Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9 · The attention family · 2026-10-09
In Vizuara's Lecture 9, Sreedath Panat covers Self-Attention with Relative Position Representations by Shaw and colleagues, Ashish Vaswani among them. A learned vector chosen by the distance between query and key is added to the key, and optionally to the value, never the query, since the query is the reference point. Distances are clipped at plus or minus two, so his example needs only five learned vectors, for -2, -1, 0, +1 and +2, whatever the context length. The idea is that nearby words matter more for meaning than absolute place in the window.
“So the relative position between the words plays a big role in how humans understand the meaning of a sentence.” Sreedath Panat · at 48:20 — open in the transcript →
Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →