What is T5's relative position bias and how is it different from a position embedding?

Sreedath Panat explains that the 2019 T5 paper, named for Text-to-Text Transfer Transformer, adds a learned scalar to the query-key dot product instead of adding a vector anywhere. Each distance maps to one of 32 buckets: distances up to 7 get their own bucket, wider ones share log-spaced buckets, and anything 91 or more away falls in the last. Each attention head learns its own 32 numbers, so 10 heads mean 320. In his example, a score of 0.9 plus a bias of -0.8 becomes 0.1.

• Buckets 0 to 15 cover keys before the query and 17 to 31 keys after; distance zero uses bucket 0, so bucket 16 is unused.
• Distances of 8 to 11 share a single bucket in his table.

▸In their words

“It rather introduces a bias score which is just a scalar.” Sreedath Panat · at 1:07:56 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode