Why is attention quadratic in the number of tokens, according to the Vizuara attention lectures?

The Vizuara instructor explains that with n tokens, every token acts as a query and every token as a key, and a dot product is taken between each query and each key, giving n times n, or n squared, operations. Sub-quadratic mechanisms such as sparse attention bring the exponent below 2, which he calls essential for long inputs like video with many frames.

Sub-quadratic cost is bucket four in Vizuara's seven-bucket scheme.
Modules four and five of the series cover sparse attention, Kimi Delta attention and related designs that cut the complexity.

▸In their words

“If you are implementing attention with n squared complexity, the scale of computation that is required will be something which no machine can handle.” Instructor · at 23:50 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-01). Read the whole episode summary →

▸More from this episode

Q1What are the seven buckets of attention mechanisms in Vizuara's attention family series?Q2How many types of attention mechanisms are there, according to the Vizuara attention lectures?Q3How does Vizuara's attention family series compare attention in Llama 3, DeepSeek V3 and Kimi Linear?Q5Is flash attention a new attention architecture or an implementation trick, per Vizuara's lectures?Q6What does Vizuara's attention family lecture series cover, and in what order?Q7Do you need to know transformers before starting Vizuara's attention family lectures?

Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →