What are the seven buckets of attention mechanisms in Vizuara's attention family series?

Vizuara's Dr. Sreedath Panat sorts every attention variant into seven buckets: scoring (how the raw attention score is computed), connectivity (which tokens may attend to which), heads and KV cache, sub-quadratic cost, GPU implementation, position information, and domain (text, image, video, audio or graph). Two mechanisms can match in some buckets and differ in others, and the series maps every concept it covers back onto these seven.

• Panat groups the buckets by goal: scoring and connectivity change what is computed, heads and sub-quadratic designs cut compute and memory cost, and kernels such as flash attention change implementation.
• Position and input type form the final group, covering how token order is injected and whether the input is text, images, video or graphs.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-01). Read the whole episode summary →

▸More from this episode