How do learned absolute position embeddings work in GPT-2?

Sreedath Panat explains that GPT-2 keeps two lookup tables learned during training: a token table of about 50,257 rows by 768 columns and a position table of 1,024 rows by 768 columns. The position table has 1,024 rows, not 50,257, because GPT-2 processes at most 1,024 tokens at once. A token's row from the first table and its position's row from the second are added element by element. The weakness is that a model stretched to an 1,100-token window has no learned vector for slots 1,024 to 1,099.

• BERT used the same learned absolute approach, added at the input embedding.
• Panat calls the fixed window the limitation sinusoidal embeddings were meant to remove.

▸In their words

“This was one of the limitations of gpt2.” Sreedath Panat · at 26:31 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode