How do sinusoidal position embeddings work?

Vizuara's Lecture 9 walks through the formula from the 2017 Attention Is All You Need paper: even dimensions take the sine, and odd dimensions the cosine, of the position divided by 10,000 raised to 2i over d. Each sine-cosine pair shares one frequency, and frequencies drop as dimensions rise. Sreedath Panat works it by hand for position 2 in 8 dimensions: dimension 0 is sin(2), about 0.91, and dimension 1 is cos(2), about -0.42. Because it is a formula, not a table, it gives a vector for any position, however long the context.

• With d of 8, the divisors for successive dimension pairs are 1, 10, 100 and 1,000.
• Each sine-cosine pair squares and sums to 1, a check Panat runs on his table.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-10-09). Read the whole episode summary →

▸More from this episode