The attention family · Vizuara · Transcript & summary by WhipScribe
A slow, numbers-first walk through the Q, K, V formula that turns the transformer's most quoted equation into arithmetic you can check by hand.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“Query is the token which is attending and key is the token that is being attended to.” Sreedath Panat · at 13:26 — open in the transcript →
“So we can't really rely on just the input embeddings alone to construct the attention scores.” Sreedath Panat · at 10:12 — open in the transcript →
Lecture 6 of Vizuara's The attention family moves from the simplified self-attention of Lecture 5, where the raw input embedding X stood in for every role, to the version used in real transformers. Sreedath Panat frames the whole lecture around one formula from the Attention Is All You Need paper: softmax of Q times K transpose over the square root of d, multiplied by V. Three learned matrices, W_Q, W_K and W_V, turn each embedding into a query, a key and a value before any score is computed.
The motivation is a failure case. In the sentence about a dog chasing a ball that it could not catch, the two uses of the pronoun share one embedding, so a plain dot product cannot tell which noun each refers to. With toy three-dimensional vectors, it scores 0.51 against both dog and ball. Passing only the keys through a diagonal weight matrix lifts the ball score to 0.96 while dog moves to 0.56, which shows how a projection can reshape similarity. Panat defines the query as the token doing the attending and the key as the token being attended to, and notes that the projected space can have a different size from the embedding space.
The mechanics are then worked twice: by hand for a five-token sentence projected from three to two dimensions, and in a Google Colab notebook using PyTorch with random 3-by-2 matrices over a six-token sentence. Each context vector comes out as an attention-weighted sum of value vectors rather than of raw embeddings. The scaling by the square root of d is only recapped, since an earlier lecture covered it. The weight matrices start random and are learned during LLM training, and the next lecture adds causal masking for text generation.
In Vizuara's Lecture 6, Sreedath Panat defines the query as the token that is doing the attending and the key as the token being attended to; the value is what gets mixed into the output. All three are made from the same input embedding X by multiplying it with its own learned matrix: Q = X times W_Q, K = X times W_K and V = X times W_V. Attention scores come from queries dotted with keys, and the final context vector is a weighted sum of the values.
Sreedath Panat of Vizuara argues that input embeddings ignore context: the same word always has the same vector wherever it appears. In the sentence where a dog chases a ball but it could not catch it, the first it refers to the dog and the second to the ball, yet both share one embedding, so a plain X times X-transpose score cannot treat them differently. Transforming embeddings with learned matrices before taking the dot product is what lets the scores reflect what each token should attend to.
Sreedath Panat gives it, dog and ball toy three-dimensional embeddings of (0.5, 0.5, 0.1), (0.9, 0.1, 0.1) and (0.1, 0.9, 0.1). Raw dot products give it 0.51 against both nouns. He then passes only the keys through a diagonal W_K matrix with entries 1, 2 and 1: dog becomes (0.9, 0.2, 0.1) and ball becomes (0.1, 1.8, 0.1). Now it scores 0.56 against dog and 0.96 against ball, so it attends more to ball, as the sentence requires.
Vizuara's Sreedath Panat unpacks the Attention Is All You Need formula, softmax(QK transpose over the square root of d) times V, as a weighted sum: each token's context vector is the sum over j of attention weight w_j times value vector v_j. For the word ball in a five-token sentence, softmax weights of 0.17, 0.16, 0.25, 0.10 and 0.31 multiply value vectors v1 to v5, and their sum is the context vector z5. Changing the query to token 3 gives a new set of weights and z3.
Yes. In Vizuara's Lecture 6, Sreedath Panat projects five tokens of three dimensions each through 3-by-2 weight matrices, so queries, keys and values are all two-dimensional, five rows by two columns. He says the query, key and value spaces share one dimensionality with each other, but many architectures use a size different from the input embedding space. The scaling factor follows the projected size, so this example divides scores by the square root of 2.
Sreedath Panat says nobody knows the query, key and value weight matrices in advance. They begin as random matrices and their values evolve during large language model training, and he calls finding good transformation matrices one big part of the art of training an LLM: good projections make the resulting dot products meaningful. In the lecture's Colab notebook the matrices are simply drawn at random with PyTorch to show the mechanics.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Hi, welcome back to a new lecture from the attention family lecture series. In today's lecture for the first time we will be discussing about query key and value. We will discuss the intuition behind that and we will also see how to implement it mathematically. I will give you a brief recap of what we discussed in the last lecture, but today's lecture in one sentence is purely about self-attention with trainable parameters which are query key and value weight matrices. So in the last lecture, lecture 5, what we discussed was a simplified version of self-attention which did not have separate query key value parameters. Instead all of those parameters were represented by the same input embedding x so our formula was softmax of instead of query times key transpose we had x times x
0:49transpose divided by square root of d multiplied by x in some examples in an even more simplified examples this dividing by square root of d sometimes will be eliminated so instead of using scaled dot product attention in some examples people show normal dot product attention also but this is what we've done what we had done in the last lecture and I hope you remember what this is this is the context vector Z so our goal was to convert the input embeddings into context vectors because input embeddings are basically independent of the available context so if there are embeddings are basically independent of the available context. So if there are embeddings for individual words or individual tokens, we need to change
1:30those embeddings slightly or we need to modify them slightly depending on the available context. So we realize that to do that, we have to construct attention scores and that too when we are constructing attention scores, we have to convert the input embeddings into context vector by taking the dot product between the individual embeddings take the soft max so that the attention scores are becoming attention weights and then you take a weighted sum of the same set of input vectors which are x to get the context vectors so this is what we did in a nutshell in the last lecture if you have not watched the last lecture please go ahead and watch the previous lecture on simplified self-attention that way today's lecture will make a lot more sense for you but at the end of the day we need
2:15a context vectors corresponding to every single input embeddings so input embeddings are denoted by x context vectors are denoted by z all right now when we take one more step back we are discussing about attention mechanisms in this series right and we said that in the very first introductory introductory lecture there are many different attention mechanisms probably 50 plus relatively famous attention mechanisms and we have to study them by bucketing them and this figure we will be coming back again and again coming back to again and again in the second third and fourth lectures we looked at the scoring mechanism so we discussed badano attention we discussed dot product attention and scaled dot product attention which was all about constructing attention scores and then in
3:03the last lecture we looked at the simplified self-attention where we are we discussed for the first time tokens attending to each other in fact in badano attention we had something like a cross attention where tokens from one sequence was attending to tokens from another sequence but in the last lecture we looked at self-attention for the first time and in today's lecture also we are continuing to look at the self-attention i will show you an outline or a map of where we are headed in today's lecture as well as in next lecture in uh in one more slide and in the future lectures whenever we discuss about new attention mechanisms you will learn about mechanisms that introduce multiple attention heads kv cache attention mechanisms that reduce the attention
3:46calculation complexity because we know that the complexity is a square of number of tokens so it's usually n square but there are also sub quadratic complex attention mechanisms we will discuss about attention mechanism implementation on GPU, injection of position information and then finally attention mechanisms implemented to different input domains like text image video graph etc. So in today's lecture we will learn about attention mechanism self-attention with trainable weights and in the next lecture we will learn about self-attention with masking or masked self-attention or causal self-attention. So this is where we are headed. So in the last lecture, simplified self-attention is done.
4:30We did not have query key value weight matrices, but this time we will talk about WQ, WK and WV, which are weight matrices, which will project the input embedding X into query space, key space and value space. We will also briefly discuss why are we first of all converting the input embeddings into query key and value because if we don't do that you will not understand why this is even existing in the first place. In the next lecture we will discuss about mass self-attention and then after that we will discuss about multi-head attention. So once multi-head attention is clear we can later move on to more advanced concepts like multi- head latent attention and so on group query attention etc so there are many concepts that we want to discuss further
5:11ahead in this lecture series so this lecture has three parts in the first part we will discuss why taking a raw dot products will fail when i say raw dot product i mean the raw dot product between the input embeddings because that's what we are currently doing or that's what we were doing in the last lecture we said the attention scores are nothing but x dot x transpose so this is like the same input embedding is being used for you know on the both ends of the dot product and then we will if we want to scale we will scale it by square root of d and then these cores are converted into weights using softmax so this will give us the context vectors now in the second part of so first part of the lecture is regarding what's the issue with this kind of a setup in fact this value
5:56vector was not existing it was x only which was the input embedding so instead of query key and value we had input embeddings only. So once we understand the problem with this kind of a setup we will realize the need for transforming the input vectors x into something else that is the reason why query key and value is existing. And couple of lectures ago we had discussed in detail why we are dividing by square root of d it is to make sure that the softmax cores will not become saturated so we will not be spending too much time on that but we will be implementing it and then finally we will discuss the famous formula for context vectors which was introduced in the attention is all unit paper which is softmax of query times key transpose query dot key transpose divided by
6:43square root of d multiplied by value vector so basically the context vectors are nothing but attention weighted sum of value vectors so we will we will discuss that definition or that particular english description of context vectors because this formula is something that you might find intimidating but actually it's very simple and if you truly think logically what we are trying to do here this formula will make even more sense so we will have a brief discussion regarding that also in today's lecture apart from all of this we will have a walkthrough through this Google Collab notebook in which there are I have already written bunch of blocks of code which will show you visually what happens to the context vectors in presence
7:30and absence of the query key value vectors or query key value conversions. So first let us discuss why this raw dot product is not the greatest of ideas. So x dot x transpose instead of doing this to calculate the raw attention scores we may want to do something else so first of all let's see what's the problem the problem is the following look at this sentence the dog chased the ball but it couldn't catch it so here you have two two tokens which are the same which is it the first it refers to what it refers to dog second it refers to ball right now if you look at the dot product between two tokens let's say dot product between it and dog versus dot product between it and ball okay so if you were to take the dot product between this first it and first dog, ideally the attention
8:28score should be very high, ideally, because here it is referring to the dog. On the other hand, if this it was being taken as a dot product with the dog, ideally attention score should not be very high because this it is referring to ball it is not referring to dog however will that difference be there actually in our dot product based attention setup where the input embeddings are directly used for constructing the dot product like x dot x transpose will this difference be there will there be any difference between the dot products? There won't be any difference because the input embedding x corresponding to this it and corresponding to this it, they are the same because input embeddings are independent
9:14of the position of the token, right? So when we simply take input embedding directly and dot product it with whichever token we are trying to attend to it can result in a situation where you end up having similar dot products so this is that this is one example where we are trying to construct the dot product between two different its with uh dot there could be other situation where you want the same it. So there could be one more example where I'm looking at this one particular it and it attending to dog versus it attending to ball. Obviously, the attention score in this case also need to be different, right? Because the it is paying more attention to dog and it should pay less attention to ball. But there
10:04could be situation where those two attention scores are similar. So it really depends on the input embeddings. So we can't really rely on just the input embeddings alone to construct the attention scores. We have to do something else to the input embeddings. The best way to remember this is let me just clear the screen once. So the best way to understand this example is looking at these two its. So instead of looking at it dot product with dog and it dot product with ball, look at this case, this it being dot product with dog and this it again dot product with dog clearly the attention scores they should be different but they will will not be different because it is the same token being you are you're taking the dot product between the same
10:55two pairs of tokens so obviously the attention scores in the naive dot product case will be the same now this is the main reason why people thought of you know making some changes to the input embedding so that when you take the dot product the dot product will make more sense so look at this example okay let's say my tokens are living in the embedding space with three dimensions so this is the three-dimensional vector corresponding to it this is the three dimensional vector corresponding to dog and ball okay now depending on the position of it it should pay more attention to dog in some sentences and it should pay more attention to ball in not in some sentences in some cases it should pay more attention to dog in some cases it should pay more attention to ball so
11:40let's say currently we are in the we are looking at this first, where we have to pay more attention to dog, okay? Or rather ball, we can take the second it because that's the example that is shown in the next slide. So if we are considering this it, the attention score should be highest between it and ball rather than it and dog, because here it refers to ball. So let's see how we can accomplish that. So here you can see that when you take a dot product between it and dog, the dot product is 0.51. It is 0.5 multiplied by 0.9 plus 0.5 multiplied by 0.1 plus 0.1 multiplied by 0.1. Now what about the dot product when you take the, what about the score, attention score when you take the dot product between it and ball?
12:28It and ball will also be the exactly same value. When you take the dot product between it and ball, it's 0.5 times 0.1 plus 0.5 times 0.9 plus 0.1 times 0.1. It will also be 0.51. So here, although you are expecting the attention score between these two tokens to be higher than these two tokens, that is not happening because you are taking a raw attention scores. Sorry, you're taking a raw input embeddings. So actually we can do something interesting to these input embeddings. So assume that we have a matrix which is actually called as a weight matrix and actually these empty cells are not empty those are zeros so here it is zero this is zero zero zero and zero so let's say this is the wk matrix basically key weights matrix so in this example that we have given
13:22can you imagine what is the query and what is the key? Query is the token which is attending and key is the token that is being attended to. So here it is attending to dog in the first example and it is attending to ball in the second example. So query is it in both cases and key is dog in the first case and key is ball in the second case. We are yet to see key and query in detail, but just for your understanding. So here this WK which is key weights matrix is something which transforms keys. It does not transform query, it transforms keys. So what that means is these two embeddings, so this is the original input embedding corresponding to dog and this is the original input embedding corresponding to ball, those two vectors will
14:10be transformed by this transformation matrix. And let us see how the transformation looks like. So, when the dog vector is multiplied or acted upon by this matrix, basically first of all what is the dimension of dog? Dog is 1 by 3, right dog dog is 1 by 3 right and this is 3 by 3 so the multiplication is nothing but x multiplied by wk right because 1 by 3 multiplied by 3 by 3 will give you 1 by 3 the resulting vector will have starting dimension of 1 and second dimension of 3 correct so what will be the result 0.9 multiplied by 1 so that will be 0.9 rest of the elements are zeros 0.1 multiplied by 0 plus 0.1 multiplied by 2 plus 0.1 multiplied by sorry 0.9 multiplied by 0 plus 0.1 multiplied by 2 plus 0.1 multiplied by 0 which
15:01will be 0.2 that's the second element third element would be 0.9 multiplied by 0 plus 0.1 multiplied by 0 which will be 0.2 that's the second element third element will be 0.9 multiplied by 0 plus 0.1 multiplied by 0 plus 0.1 multiplied by 1 so the last item will be 0.1 so after transformation the vector corresponding to dog got transformed into 0.9 comma 0.2 comma 0.1 originally it was 0.9 comma 0.1 comma 0.1 so the second item got changed what about ball what happens to ball when the same matrix transformation matrix is used for transforming ball the result will be 0.1 multiplied by 1 plus 0.9 multiplied by 0 plus 0.1 multiplied by 0 so first element will be 0.1 only second element would be 0.1 multiplied by 0. So first element will be 0.1 only. Second element would be 0.1 multiplied by 0 plus 0.9 multiplied by 2 plus 0.1 multiplied by 0. So that will be
15:500.9 times 2, 1.8. So second element will be 1.8. Similarly, third element would be simply 0.1 multiplied by 1, which is 0.1. So you see the vector corresponding to ball was initially 0.1 comma 0.9 comma 0.1. Now it became 0.1 comma 1.8 comma 0.1 now it became 0.1 comma 1.8 comma 0.1 okay now let's see what happens to the dot product now if we take the dot product between it and uh dog and it and ball so here we are not yet transforming it that we will do in the next upcoming slides but here we are only transforming the keys the queries we are not transforming query is the the token that is uh attending to other tokens keys are the tokens that are being attended to so in this case dog is being attended
16:31to by it ball is being attended to by it so uh the dot product between it and transformed dog will be 0.5 multiplied by 0.9 plus 0.5 multiplied by 0.2 plus 0.1 multiplied by 0.1 right so this will be 0.45 plus 0.1 plus 0.01 which is 0.56 so the dot product has changed a little bit earlier the dot product between it and the original version of dog was 0.51 now it became 0.56 but that's not the interesting part the interesting part is the next one if you look at the dot product between the original input embedding of it and the transformed embedding for ball it would be 0.1 sorry 0.5 multiplied by 0.1 which is 0.05 plus 0.5 multiplied by 1.8 which will be 0.9 plus 0.1 multiplied by 0.1 which is 0.01 and that will become 0.5 multiplied by 1.8, which will be 0.9 plus 0.1 multiplied by 0.1, which is 0.01 and that
17:27will become 0.96. Now you see it as a token is now attending, paying more attention to ball and less attention to dog because that is the case. In our case, the example sentence that we said, we said we are looking at this particular it and this it is referring not to the dog it is referring to ball which means the attention score between it and ball should be higher so here the attention score between it and ball is actually higher but which ball are we talking about are we talking about the original input embedding for ball no we are talking about the transformed input embedding for ball similarly the it also we have to transform for same reasons like there might be a case where the query also requires transformation, not just key, right?
18:14So in an ideal world, when two items are being, you know, attending to each other by taking a dot product x.x transpose, we will not use x as such here we will transform it to something like query here also we will not use x as such we will transform it to something called key so key transpose and this transformation is brought about through these matrices called as transformation matrices or wk wq wv etc. Now the whole art of I would say one big part of training the large language model is figuring out what are these good transformation matrices how do you transform your input embedding embeddings into rich query representation rich key representation and value representation because if the if the transformation matrix is good and after transformation your dot products will make
19:06more sense so in this case after step three the dot product makes more sense why because the dot product between it and dog is smaller than the dot product between it and ball which is that is how it is supposed to be in that particular example sentence all right so in short this is what I have already mentioned in general every embedding so if I if a sentence has six tokens all the tokens which we can in general denote using X is transformed into query queue key K and value V earlier in the previous slide we saw how x is transformed into k x is transformed into k using wk so basically k is equal to in the previous example it was x times wk it can be opposite also depending on the dimensionality in some cases you might multiply wk times x so it completely depends
19:59on what's the input dimension what's the expected output dimension similarly q the query will be x times wq and value will be x times wb so here what's happening to your input embeddings in one step you are converting your input embedding into query space this is a different dimensional space the dimensionality of the query space can be different in the previous example it was the same it was still a three-dimensional space but there are cases where space can be different. In the previous example, it was the same. It was still a three-dimensional space, but there are cases where it could be different. In some cases, in another case, you are converting the input embedding X into key. So key space, query space, and value space,
20:36all of them will have same dimensionality, but that dimensionality may be different from the dimensionality of the input embedding space. And the same X is also converted into value space. So instead of you know instead of doing something like softmax of x times x x dot x transpose divided by square root of d multiplied by x what is actually done and this is the context vector right the formula for producing context vector this is also from the attention is all you need paper in that paper what we have instead of this is this x is replaced with query this x is replaced with key and this x is replaced with value so we never use the input embeddings as such for for a simple reason which i showed you earlier because if you use it directly the same token
21:23may mean different things depending on what is the available context so in this equation at every every place you use transform versions of input embeddings so that is the idea behind query key value so just to recap in lecture 5 which was the last lecture we are working we were working in the raw input embedding space so the same input embedding we used for dot product to produce the attention score. So, attention score was dot product between x and x and then the output which was the context vector was nothing but this score which is converted into weights which we can call as w. So, scores when you do divide by square root of d and when you pass it through softmax you get the attention weights that attention weight multiplied by the vector themselves and you sum it up that is the final context vector whereas in current case what we
22:09do is we before we do the dot product we convert the embeddings into uh query space key space and value space so this is the embedding converted to query this is the embedding converted to key so the resulting queries and keys are what we do the dot product with and that will give us the attention scores attention scores are converted into attention weights by dividing by square root of d and by taking softmax and then that resulting quantity we will multiply with uh the value vectors which is also resulting from the input embeddings input embeddings are converted into values value vectors and the attention weighted sum of these value vectors is our final context vector so when we say output here essentially what we are talking about
22:50is context vectors so in both in both last lecture and current lecture we are doing the more or less the same thing we are somehow figuring out what's the best way to convert the input embeddings to context vectors that is what we are doing and we realize that just taking direct dot product between the tokens is not the best idea we have to convert the token into a different space and that will help us produce richer representations right so now if i say the context vector z is attention weighted sum of value vectors do you understand what it means so this is the formula right softmax of query time query dot key transpose divided by square root of D multiplied by W multiplied by V attention weighted sum of value what does that mean so
23:37element wise this is how the multiplication with value will look like W of J multiplied by V of say what is v of j v of j is value vector number j w of j is the corresponding attention weight and you sum this across basically you will be doing summation of sigma j equal to 1 2 let's say 5 if there are 5 tokens you will do j equal to 1 to 5 so attention weighted sum of value vectors is what the context vector is, right. And when you, when somebody now says this statement, you should be able to visualize this equation. So, this is the value vector, you are, and this is the attention weights, you are multiplying attention weight with the value vector and you are summing those value vectors to get the context vector, okay.
24:25All right. Now, let us do a quick query key value calculation by hand for the example sentence that we will choose. The sentence is the dog chased the ball, okay. The dog chased the ball. We have 1, 2, 3, 4, 5, five tokens and every token is living in a three dimensional space now let's say these are our query key and value matrices transformation matrices wq query weights matrix key weights matrix and value weights matrix you should be already be able to imagine what happens to the the tokens after conversions do you think the tokens will still remain in the three-dimensional space after conversion because conversion is x times wq or x times wk and x is 1 by 3 and wq wk etc are 3 by 2 so the resulting vector will be of 1 by 2
25:21dimensionality right so the dimensionality of the space is different dimensionality of the query space key space and value space is two dimensional whereas the original input embeddings are living in three dimensional space so here you can see that the dimensionality of the space is changing after you project your input embeddings into query space key space and value space respectively okay so just note this this can happen like there are many architectures where the query key and value space are not necessarily of the same dimensionality as that of the original input embedding. So 5 by 3 because you have 5 tokens 1, 2, 3, 4, 5, the dog, the dog chase the ball, you have 5 tokens each of them are of 3 dimensions and transformation matrix is 3 by 2 and resulting
26:04query is 5 by 2, key is 5 by 2, value is three by two and resulting query is five by two key is five by two value is five by two what does five by two means you have if you have five input tokens you have five queries if you have five input tokens you have five keys and five values original input type tokens were in three dimensional space resulting query key and value they are all in two dimensional space there is a small mistake in the diagram here actually here there are only four rows actually there should be one more row fifth row should be there okay that's that's just a simple uh image rendering mistake but uh that is what happens to the input embeddings now the keys queries keys and values which are resulting from our input embeddings will look
26:40like this we can actually do a proper calculation um in the google collab notebook but here i'm just showing you for initially before we run the collab notebook so now this is the query corresponding to the query corresponding to dog chased the ball right these are all the queries these are all the keys corresponding to the same set of tokens and these are the values corresponding to the same set of tokens now once we have the query key and value let's say i want to calculate the context vector corresponding to one particular token because we are essentially converting tokens to context vectors so in our case we have five tokens right five total tokens which means we will have five context vectors five z's we had five x's
27:27five x's meaning x is the input embedding we had we had five of them the dog chased the ball which means we will also have five context vectors so let's try to understand how one of these context vectors are calculated so let's say we are trying to calculate the context vector associated with ball so when we say we are calculating context vector associated with ball, ball is what? Is it query key or value? Ball is the query. Ball is the query against every key. So when we take dot product, ball will act as the query and every token in the same sequence will act as the keys. So first what we do is is we will calculate the dot product between q of ball so let's actually number them this is one two three four five right so q
28:16we are always taking q5 dot k1 because you will convert the token corresponding to the into key and query. And here also you are doing Q5 times K2. Here you will do Q5 not times dot product Q5 dot with K3, Q5 dot with K4 and in the last case it will be Q5 dot with K5. And this dot product will give us the attention scores, where are the attention scores and you will also divide by square root of the dimensionality, but these are the raw attention scores 2.89, 2.82, 3.42 etcetera, these are the attention scores which are dot products between your queries and keys. Now, these dot products the raw attention scores you divide by square root of 2, why are we why are you dividing by square root of 2, because 2 is the dimensionality of the
29:02query key value space in this specific example so after dividing by square root of 2 you get the scaled attention scores then these scaled attention scores you convert into attention weights by passing through softmax so you now have 0.17 0.16 0.25 0.10 and 0.31 now how do we get the context vector context vector is nothing but 0.31. Now how do we get the context vector? Context vector is nothing but 0.17 times v1 plus 0.16 times v2 plus 0.25 times v3 plus 0.10 times v4 plus 0.31 times v5. So this is the weightage given to v1, this is the weightage given to v2, this is the weightage given to V2, this is the weightage given to V3, weightage given to V4 and weightage given to V5. So, you are essentially adding up V1, V2, V3, V4 and V5, these are value vectors, but
29:57you are not directly adding them, you are adding each of them by having a pre factor or a weight, what are those weights? Those are the attention weights. So attention, this is the attention weighted value vector. This is also attention weighted value vector. This is also attention weighted value vector. You are taking the sum of those attention weighted value vectors to get the context vector. And which context vector will this give us? Is it Z1, Z2, Z3, Z4 or Z5? This is going to give us the context vector corresponding to query number 5. So this will give us Z5. Similarly, if you wanted, let's say Z3 is what you wanted. What is the change that you would have made here? If you wanted Z3, instead of Q5 over here you will use q3 you will do q3.k1 here q3.k2 q3.k3
30:51q3.k4 and q3.k5 that will give you another set of attention raw attention scores which you will do by divide by square root of d which will give you a new set of weighted sorry scaled attention scores which you will convert through softmax into attention weights which will give you a new set of weighted sorry scaled attention scores which you will convert through softmax into attention weights which will give you new weights and those attention weights again you will multiply with v1 v2 v3 v4 and v5 respectively you will add them to get what in that case you will not get v z5 you will get z or z3 so this is how the context vectors can be calculated for the token number one two three four or five right and
31:27all of this can be represented in the in a very compact manner using matrices but that is in my opinion slightly more difficult to sometimes visualize than this so this is what i mentioned in the previous slide the context vector of ball is a mixture of values, values meaning a mixture of value vectors v, not the mixture of embedding x, okay. So v of the which is v1 multiplied by 0.17 plus v2 multiplied by 0.16 plus v3 multiplied by 0.25 plus v4 multiplied by 0.1 plus v5 multiplied by 0.31. The sum of those will give you the z corresponding to ball, which I can also call as z5. And that will obviously be in a two-dimensional space because our query key and value vectors are all in the two-dimensional space. So when you are doing a sum, weighted sum or
32:17normal sum, whatever it is, when you add vectors, the resulting vector is living in the same space. So it will live in the two-dimensional space. So, now context vector corresponding to ball is two-dimensional whereas the original input embedding corresponding to ball was three-dimensional in our specific example. So, this we had already discussed in the previous lecture the need for dividing by square root of d. So, I am not going to spend too much time in this lecture because it is now very clear. If it is not clear why, just go to the lecture number 3 where we discussed about the dot product attention and scaled dot product attention. If you watch it, you will completely understand why we are dividing by square root of d. But I can
32:58quickly give you a recap. What happens is when you take dot product between q and k, q dot k transpose and let us say the query is a random you know initialize you initialize the query as a random vector, key also you initialize as a random vector and let us say this is the dimensionality of the embedding space. This dot product basically scales by proportional to square root of d. So when what happens is when you scale the attention scores as a square root of D. So, when what happens is when you scale the attention scores as a square root of D, you may end up the the raw attention scores what will happen is sometimes you may have attention scores like in one case let us say attention score is 5 comma
33:366 raw attention score let us say. In another case let us say the raw attention score is 500 comma 505 okay after you convert this into a softmax format okay what do you think will happen in which case do you think you will get a saturated softmax like this so saturated softmax meaning one item is like this another item is completely dominating so let's say this is 0.1 and this is let's say 0.9 because this is close to a saturated softmax in which case is it in this case or this case you will get a saturated looking softmax you will get a saturated looking softmax in this case you know why because softmax formula is e raised to x divided by Sigma e raised to x or e raised to x i divided by Sigma e raised to x divided by sigma e raised to x or e raised to xi divided by sigma e raised to xi
34:26so in this case the softmax will be e raised to 505 divided by e raised to 505 plus e raised to 500 that will be one of the numbers the other number would be e raised to 500 divided by e raised to 505 divided by plus e raised to 500. so if you compare e raise to 505 versus e raise to 500, this is a very big number compared to this number. On the other hand, in this case, it is a comparison between e raise to 5 versus e raise to 6. Yes, of course, e raise to 6 is bigger than e raise to 5, but not by that much of a margin. So this is the reason why the relative size difference between e raised to 5 and e raised to 6 versus e raised to 505 versus e raised to 500. That is the reason why we have to scale down the raw attention scores by dividing by attention
35:16scores by square root of d. So what happens is in that case the magnitude of query dot key transpose on an average will stay as a flat line. Basically magnitude of query dot key transpose on an average will stay as a flat line. Basically, magnitude of query dot key transpose will not depend on the dimensionality. It will not, I mean, when we plot query and key transpose, we are not saying that all the query and key transpose values will be the same. No, that's not what we are saying. What we are saying is query dot key transpose value that will be independent of the dimensionality. Even if dimensionality becomes 5000 it will not affect the magnitude because we are dividing by square root of d this we had
35:51discussed in detail in the two lectures ago so please go ahead and have a look at it so finally this is attention is all you need papers famous formula attention of query key value equal to softmax of query dot key transpose divided by square root of d multiplied by value from here on you should be very comfortable thinking about this formula what is this formula by the way this is context vectors this is the formula for calculating context vectors so context vectors as a function of query comma key comma value this is the i like to call it call this as the formula for obtaining context vectors as a function of query comma key comma value. This is the, I like to call it, call this as the formula for obtaining context vectors
36:26as a function of query key value. The other, as a function of query key value, we get context vectors using this formula. And by the way, how did you get Q? Q you got by doing X times WQ. How did you get K? You got K by doing X times WK. And you are taking the transpose of that. And then you are doing square root of D. and you are taking the transpose of that and then you are doing square root of d and you are taking softmax i'll just call it as soft softmax multiplied by value how did you get value x times wv in the simplified self-attention in the previous lecture this wq wk and wv was non-existent that was a difference between that's the difference between the previous lecture and the current lecture now i think it's a good time to go to the google collab notebook and do
37:11our implementation but just just remember this picture this picture is telling what the same story what the formula is telling you this is the input embeddings you have five embeddings the dog chased the ball they are living in three-dimensional space you project them into query space key space and value space the picture has a mistake actually here i'm only showing three rows but actually it's not three rows it's five rows dimensionality is two but there are five rows okay so just don't be mistaken by the this picture ideally the picture should have been something like this so it's just a rendering issue in the picture, but apart from that you know there should be 5 rows in all of these query key and value matrices.
37:48Matrices as in they are the collection of vectors and then query dot key transpose will give you attention scores, attention scores are divided by square root of d and taken softmax to give you attention weights and then those attention weights you multiply with v to get the final context vectors. So, this is the final z context vectors, you have five context vectors living in two-dimensional space which you obtained from the five input embeddings which was living in three-dimensional space okay so now let us move on to our google collab notebook so uh it's very simple i'm not going to walk you through line by line it's very very simple but i'll give you the gist of what we are trying to do here.
38:27So the same vector embeddings you saw earlier, this is the vector embedding corresponding to dog, this is corresponding to ball and this is corresponding to it. And if you take the raw embeddings and if you try to reproduce the dot product, the dot product between the embedding corresponding to it and dog is 0.51, it and ball is also 0.51, but in reality the it that we are talking about right now is the second it the dog chased the ball but it could not catch it so we are talking about this particular it that it ideally should have more attention towards ball and lesser attention towards dog because this it is not talking about ball so but here the attention scores are the same so the fix is nothing but
39:05introducing a wk matrix actually you know this is not a vector this is a matrix here we are already defining it as a diagonal matrix so torch that dot diag means you just need to define the diagonal elements okay so these three will become the diagonal elements so after transforming our dog and ball using this diagonal matrix the dog will become something like actually we can print print the dog right so print um wk multiplied by ball. So we can see after transformation how these vectors also look like. They will be three dimensional because here the transformation matrix is three by three, it is not two dimensional. So this is the transformation of the dot, this is the transformation of ball and this
40:03example we already saw. So after taking the now the new when the new dot product is taken between the original embedding and the the transformed dog and the original embedding and transform ball the attention score is higher between it and ball okay now similarly we can have a set of input embedding. So these are our five input embedding actually six input embeddings green big and Work for it. So this is the sentence So you have six tokens and as a result you have six input embeddings, by the way How do you get the input embeddings input embeddings you get from a embedding matrix or embedding table that we have not discussed in any lectures because When we are talking about attention scores, we are already assuming that we have not discussed in any lectures because when we are talking about attention scores we are already assuming that we have input embeddings
40:47available so these are the original input embeddings corresponding to these tokens and what we have is we have the query weights matrix key weights matrix and value based matrix in our current case we are defining them as random weights random matrices okay so torch dot ranch torch dot rant now we can print them so this is the query weights matrix similarly we can print key weights and value weights so wk w key and w value value. So these are the wq, wk and wv, they are 3 by 2 which means you have 3, you will result the resulting context vectors will be of dimension sorry the resulting query and key and value projections will be of dimension sorry the resulting query and key and value projections will be of dimension um so this is 1 by 3 this is 3 by 2 which means result will be 1 by 2 so you are
41:50projecting from 3 dimensions into 2 dimensions then uh let's do let's take one particular query the query token that we are interested in picking is this one so we will say this is the query which we want to convert into context vector okay so this is the item number one right index number one so the query x2 is inputs of one so now how do you convert x2 into query you convert x2 into query by multiplying it with the w query vector you convert it into key by multiplying with the w key vector sorry not w key vector the w key matrix and you convert the same into value using w value matrix and attention scores can be calculated by taking the dot product between the current query and all the keys right because uh you have you just so this is just that
42:40here we are we are just printing the attention score between query number two and key number two, but we need attention score between all of them, but this is just a sample. Okay. So we're just scoring it against itself here, but we actually need to take the attention scores where we are taking the dot product between the query number two and key number one, query number two and key number two, query number two and key number three and so on so now the queries are inputs inputs is this this matrix this is the inputs matrix so we get all the queries by multiplying input matrix with w query all the keys by doing the same with w key and similarly values by multiplying with w value attention scores are the product of queries multiplied by keys
43:25transpose so queries this is matrix multiplication symbol in python key transpose is keys dot t so this is basically the transpose operation and then we print the attention scores we can actually print the actual attention scores not just the shape so let's just print this. So, now these are the attention scores. We have 6 by 6. We have 6 queries and 6 keys. Then we scale by square root of d. So, dimensionality of the keys is the dimensionality of the basically the total number of columns in our weights matrix. The number of columns is basically 2. So, that is the dimensionality and attention weights are nothing but attention score divided by square root of d then you take softmax and when you take softmax you are taking it along rows you want the rows to add up to one not the column
44:19so that's why here you give the additional parameter dimension equal to minus 1 and then we can just print the attention weights so attention weights corresponding to big is this row and the we can also see what is the sum of every single rows so every single row should add up to one so this this line is basically printing that i don't even want to print this so i can just remove it for simplicity so now these are the attention weights corresponding to the token query so this is the dot product between q1 and sorry q2 and k1 this is between q2 and k2 so when i say q2 i mean query number two which is which is this guy inputs of one okay inputs of one is query number two so this is the dot product between q q2 and k1 this is between q2 and k2 q2 k3 q2 dot k4
45:16q2 dot k5 and q2 dot k6 it's not just dot product you take dot product you divide by square root of d and then takes off max and this is the final result now how do you get context vectors context vector is nothing but this times v1 plus this times v2 plus this times v3 plus this times v4 plus this times v5 plus this times v6 that is the context vector um so let's now do that let's calculate the context vector so now this is the so this this cell i'm deliberately skipping because it is just showing you with and without division with square root of d. This we had already discussed in detail in the past lecture. So I am not going to spend time on this. But now the context vectors are nothing but these attention weights multiplied by values.
46:04So attention weights multiplied by values. You, attention weights multiplied by values. You could also do something like this, context vector number, context vector for 2 is attention weights of 1 multiplied by values and here you can just print context vex 2. This will print the same thing. So instead what we are doing here is, if I go back, we are calculating all the context vectors by taking all the attention weights and all the values and then out of all the context vectors we are only printing the context vector at the index number one. The context vector at the index number one is the one corresponding to the token big. So that is 0.2274 comma 0.7362. Then I think this we have, this we will not be doing right now.
46:52We can discuss the actual python implementation of the self-attention a bit later. So actually you know what, I don't want to keep these cells right now. This we can implement in future lectures. I am just going to keep these cells right now, this we can implement in future lectures, I am just going to delete this, because when I share the this code block with you, I do not want you to feel overwhelmed, alright. So that is all I just wanted to show you in today's lecture. So I hope you get a pretty good idea now how context vectors are resulting from the attention of QKV formula from our attention is all you need paper. So that brings an end to today's lecture. Now one thing we did not discuss here is how exactly the WQ, WK and WV values are obtained,
47:37I mean those matrices are obtained. Those matrices are, they will start as random matrices, but their values will evolve as you proceed with the large language model training. Okay. So that it's not something which somebody knows in advance. Nobody knows it in advance. So that's one takeaway that I wanted to mention. Now, in the next lecture, I want to discuss one important thing, which is masked self attention. So in the self attentionention which we discussed today every query was allowed to attend to every other query but that is not true when it comes to models that generate text when a text generation model is being trained the current query let's say query number two should not be allowed to attend to key number five or key number four
48:23current query should only be allowed to attend to keys up until itself key number 4. Current query should only be allowed to attend to keys up until itself. So if you are looking at Q3, Q3 should attend to K3, K2 and K1. It should not attend to K4 and K5. We will discuss the reasoning behind this and reasoning behind masked self-attention and why is it needed for generating text in the next lecture. But that is what we will cover in the next lecture but that is what we will cover in the next lecture masked self-attention so if you go back to one of the figures that i showed you in the initial slides i think it was slide number four or something yeah if you look at a slide number this slide we in today's lecture every tokens were allowed to attend to every other
49:03token we are still in the connectivity bucket. And in the future lecture, we will see why every tokens are not allowed to attend to every other tokens, especially in the case of mass self-attention. So that's all for today's lecture. Let's wind up. We will meet again in the next lecture. Take care until then.
← Part 5Simplified self attention | The attention family series - Lecture 5Part 7 →Causal attention (masked self attention) | The attention family series - Lecture 7