podcasts / The attention family / episode / question
Question

Why does every position get a unique sinusoidal embedding if sine waves repeat?

Answered from Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9 · The attention family · 2026-10-09

Sreedath Panat of Vizuara explains that only the low dimensions repeat quickly. In his example, dimension 0 repeats about every 6.3 positions and dimension 2 about every 62, while the highest dimensions change so slowly they never repeat inside a realistic context. Two positions may share values in a few fast dimensions, but the whole vector still differs, and the chance of a full repeat shrinks as dimensions grow. He shows it with a heatmap of 32 dimensions across 48 positions.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode

Q1Why do transformers need position embeddings, according to Vizuara?Q2Does adding a position embedding change a word's meaning in a transformer?Q3How do learned absolute position embeddings work in GPT-2?Q4How do sinusoidal position embeddings work?Q6How does Shaw et al.'s relative position representation work?Q7What is T5's relative position bias and how is it different from a position embedding?Q8What happens to a small language model trained without position embeddings?
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →