Does adding a position embedding change a word's meaning in a transformer?

Sreedath Panat of Vizuara argues it almost never does, because the embedding space is extremely sparse. GPT-2's roughly 52,000 tokens live in 768 dimensions, while filling that space even coarsely, at 10 steps per axis, would take 10 to the power 768 vectors. A position vector can be as large as the word vector, yet the sum points somewhere no other word does, so 'cat' plus position stays closest to 'cat', not 'dog'.

Panat cites an experiment tied to the GPT work: across about 2,000 words and 128 positions, the shifted vector stayed nearest its original word in 99.999% of cases.
The same analysis found position vectors pointing mainly into about 10 of the 768 dimensions, where only about 0.4% of word-embedding spread lies.

▸In their words

“The space, the embedding space is very, very sparse.” Sreedath Panat · at 14:33 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode

Q1Why do transformers need position embeddings, according to Vizuara?Q3How do learned absolute position embeddings work in GPT-2?Q4How do sinusoidal position embeddings work?Q5Why does every position get a unique sinusoidal embedding if sine waves repeat?Q6How does Shaw et al.'s relative position representation work?Q7What is T5's relative position bias and how is it different from a position embedding?Q8What happens to a small language model trained without position embeddings?

Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →