Why do transformers need position embeddings, according to Vizuara?

In Vizuara's Lecture 9, Sreedath Panat uses two sentences made of the same four tokens: 'all dogs are animals' and 'all animals are dogs'. Without position information, 'dogs' and 'animals' get the same embeddings in both sentences, so they project to the same queries and keys and the dot product between them is the same number, even though the word order, and the meaning, differ. Attention on its own ignores order; position information has to be injected so the two scores come out different.

• Panat assumes one word equals one token for the example, so dogs sits at a different index in each sentence.
• Once a position vector is added, the query-key product expands into extra terms that involve position, which is what separates the two scores.

▸In their words

“Clearly the order in which tokens are arranged in a language matters.” Sreedath Panat · at 6:54 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode