How are the W_Q, W_K and W_V weight matrices learned in an LLM?

Sreedath Panat says nobody knows the query, key and value weight matrices in advance. They begin as random matrices and their values evolve during large language model training, and he calls finding good transformation matrices one big part of the art of training an LLM: good projections make the resulting dot products meaningful. In the lecture's Colab notebook the matrices are simply drawn at random with PyTorch to show the mechanics.

• Lecture 7 of the series covers masked, or causal, self-attention, where a query may only attend to keys up to its own position.
• Later lectures in The attention family move on to multi-head attention, multi-head latent attention, grouped query attention and KV cache.

Source: our own transcription of the episode. The full conversation is on The attention family (2026-09-18). Read the whole episode summary →

▸More from this episode