Answered from Q, K, V intuition - self attention with trainable weights | The attention family series - Lecture 6 · The attention family · 2026-09-18
Yes. In Vizuara's Lecture 6, Sreedath Panat projects five tokens of three dimensions each through 3-by-2 weight matrices, so queries, keys and values are all two-dimensional, five rows by two columns. He says the query, key and value spaces share one dimensionality with each other, but many architectures use a size different from the input embedding space. The scaling factor follows the projected size, so this example divides scores by the square root of 2.
Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-18). Read the whole episode summary →