Answered from Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9 · The attention family · 2026-10-09
Sreedath Panat closes Vizuara's Lecture 9 with a tiny GPT-style model trained on the TinyStories dataset in Google Colab, once with and once without position embeddings. Test loss is lower with position information, though the gap is smaller than expected, since the network learns some things without it. The generated stories differ more: without position the output is close to a list of words, while with position it shows some sense, such as a girl deciding to go to the park, though still ungrammatical after very little training.
“When the position information is injected the output becomes more sensible.” Sreedath Panat · at 1:12:28 — open in the transcript →
Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →