Causal attention (masked self attention) | The attention family series - Lecture 7

0:01Hi, welcome back to a new lecture from the Attention Family Lecture Series. Today's lecture is focused on Masked Self-Attention, also called as Causal Self-Attention or Causal Attention. This is a very important concept in text generation models and we will first try to understand the intuition behind masking and then we will discuss the mathematical way of implementing masking. The concept itself is very simple. you are already aware of this you can feel free to fast forward the lecture or if you are coming across this concept for the first time or if you want to refresh it please feel free to continue ahead with me. I will also be showing you a simple code implementation of mask self attention and the mathematical construct behind that.

0:46So here is a quick recap of what we covered in the last lecture. We are covering various aspects about attention mechanisms so that once we are fully clear about self-attention with trainable weights, with masking, with multiple heads, we are good to go ahead and learn about many other different attention mechanisms. So in the last lecture, we looked at, for the first time, the concept of projecting our input token into query space, key space, and value space using query-based metrics, key-based metrics, and value-based metrics. And this query key and value is used for calculating the attention score. In fact, query and key is used for calculating attention score and attention weight. And then you take softmax of that and then you multiply with

1:30value to get the context vectors. So if you have originally five input vectors, they will be converted into five context vectors. This diagram has a small mistake here. Query, it seems like there are only three rows, but actually there should be five rows because if you have five tokens so here you can see one two three four five you have five tokens each living in three dimensional space it is being projected into two dimensional space of query key and value which means you should have five vectors which are living in two dimensional space so this the image rendering has an issue but other than that the concept i hope is very clear very clear for you so this was the formula attention of qkv which is aka context vectors

2:11is equal to softmax of query dot key transpose divided by square root of d4 scaling multiplied by value so this is attention weighted sum of value vectors it's a in one line that's how we can recap but remember the sentence attention weighted sum of value vectors. It's a in one line that's how we can recap but remember the sentence attention weighted sum of value vectors because that sentence we will look back at when we are discussing mass attention or causal attention. So now let's come back to this diagram that we have been referring to again and again. So currently we are in this connectivity section connectivity meaning which tokens are supposed to attend to what tokens so set mass attention or

2:48Causal attention is a classic example of you not acting you not allowing the queries to attend to certain keys in construction of attention scores Once we are done with this in the next lecture. We will move on to the multi head attention concept but by the end of today's lecture hopefully a good amount of discussion about the attention score calculation is done and a good amount of discussion about the connectivity is also done however there are future attention mechanisms where we will come back to this connectivity idea where what tokens are allowed to which token that again will come into picture in cross attention it will again come into picture in axial attention many other attention types

3:30especially used in models that are not necessarily dealing with text when you are looking at images cross attending to each other or image and text sequence token sequences attending to each other there are other concepts which will prompt us to again come back to this connectivity idea but for the time being we are currently focused on how tokens attend to each other so current lecture although it's about mass self-attention we are not discussing about multi-head attention we are still discussing about a single attention head however you know that in gpt like models it has multiple attention heads and each head is having its own output and later you concatenate those outputs to get back the final set of context vectors. But we have not discussed this till in this

4:15lecture series. If you are familiar with this, of course you will understand what I am talking about. But otherwise, you can wait until the next lecture to see how multi-head attention works in practice. So here we are at masked self-attention. This was done two lectures ago. Simplified self-attention without trainable weights. Self-attention with trainable weights was done in the last lecture. In the future lecture, we will look at the multi-head attention. So I hope the lay of the land is very clear for you. So today's lecture, I have split into three parts as usual so all lectures we generally have this three part structure so first we will discuss why is it very much required to hide the future so when i say hide the future it feels a little bit abstract although you can

4:58imagine something like okay we are doing something with respect to hiding the future tokens but you may still have questions like what do we mean by future tokens here and how do you tie that concept mathematically back into our equations although it's all very simple if i ask a few questions it may prompt some thoughts in you so i will i will discuss all of that in the upcoming slides and then what are the different ways in which we can implement masking because it's not like there is only one way to do masking there are multiple ways to do masking what are the pros and cons which we can implement masking? Because it's not like there is only one way to do masking. There are multiple ways to do masking.

5:29What are the pros and cons of each approaches? And what is the way in which masking is implemented currently? And third, we will also discuss the concept of dropout, which is very important when you are training a GPT-like model so that there is no overfitting. But here, when we are talking about dropout, it's specifically with respect to attention weights. So attention weights matrix which is a square matrix there we will implement dropout and we will do something with the remaining elements which are not dropped out so we will have a brief discussion about the dropout itself and we will see the whole pipeline how to implement it in code so I will give you a proper code walkthrough just like the past lectures

6:03so these are the three parts of today's lecture. So in the first part, let us discuss about why do we hide certain queries and keys from attending to each other and what do we even mean by hiding? So let's see that. So let's say we are in the inference. So inference meaning we are, we have the trained LLM and we are trying to produce the text, which means we are trying to produce iteratively next token after next token after the next token and say this is what was written so far dream big and and let's say the next token that is going to be predicted is work so there is a difference when you think about the LLM training and LLM inference because during LLM training you have everything available to you you have let's say this is the sentence dream big and work for it so let's say that's the training you have everything available to you. You have, let's say this is the sentence, dream big and work for it.

6:47So let's say that's the sentence. You have the whole sentence available to you during training. But think about this. Let's say currently you are considering only these three tokens and you are trying to perform the next token prediction as if you are in inference. Because during inference you don't have access to any other tokens tokens you only have access to tokens that are produced up until that point so let's say you have only these tokens available dream big and what do you think the context vector is context vector as always it is attention weighted sum of value vectors which value vectors by the way because here technically if you were in training, you have all these 1, 2, 3, 4, 5, 6 tokens available.

7:28Of course more tokens are there, but in this constraint example, you have 6 tokens available. If you are in training, because during training you have all the data available, during inference you don't have. So, if you were training a large language model, which is where this masking concept is coming into picture in the first place place even if you are considering these three tokens to predict the next token called work you already have access to all the tokens does it mean that you should use the value vectors corresponding to all the tokens to produce the context vector because if that was the case you you have how many value vectors 1, 2, 3, 4, 5, 6. So in the current context window, you may have six value vectors.

8:11However, you are trying to predict the next token called work, which means when you are constructing the value vector, is it the attention weighted sum of all the six value vectors or is it the attention weighted sum of these three six value vectors or is it the attention weighted sum of these three value vectors, the value vectors corresponding to these three tokens, which one would it be? Just think about it for a second. If you train your model such that it is, if for predicting the next token work, if you are considering the attention weighted sum of all these six value vectors, what you are essentially doing is before even the token for and before even the token it is being predicted, you are somehow

8:51accessing the value vector corresponding to them. That will never happen in inference, right? During inference, you don't have the value vectors for the future tokens because you haven't produced them yet. You only have value vectors corresponding to tokens you produced up until that point. So if that is the behavior that you expect during inference, during training also you should inject that behavior. The behavior during training, how can the behavior be different when suddenly you are doing inference versus when you are doing training. So during training also, although you have the whole text available to you during training, when you are predicting the token corresponding to work and if you want to create the context vector corresponding to that token prediction,

9:32you should only consider attention weighted sum of which value vectors, all the value vectors which come which came up until that point. So that is the core principle behind mask self-attention. I will show you where masking is coming here but this is the core principle behind mask self-attention. I will show you where masking is coming here. But this is the core principle. When we say attention weighted sum of value vectors, it is only attention weighted sum of value vectors which came up until that point in the prediction. Only those are used for predicting the context vector or only those are used for constructing the context vector. And that context vector can be used for saying what is the next token.

10:07So that is the first thing. Now let us see what does this mean mathematically. So whenever we discuss about mass self-attention, the mask is nothing but an upper triangular mask. So this is what this is, these are attention weights matrix, which is a square matrix. And when I say upper triangular, all these elements, all these upper triangular elements will be basically be zeros. Okay. First of all, that is mathematically what it means by mass self-attention. We will see how this is accomplished from the previous step of attention scores, how these attention weights are obtained, that we will see. Another thing that you should observe is still the principle that every row should sum up to 1, that still holds.

10:57This row will sum up to 1, this row will sum up to 1, every row will still sum up to 1. And now we will see why is why is it this structure we will come to the mathematical implementation of this but why is this upper upper triangular structure making sense let's say i have only the current token which is dream and i am trying to predict the next token which is big if that is what i am trying to do do i have the value vectors corresponding to big and work for it? Do I have those five value vectors available for me? I don't have, right? So what that means is the attention scores, which is going to be multiplied with those five value vectors should be zeros so that even if I have those five value vectors even if I have all the six value

11:45vectors readily available during training because during training I have all the six tokens available dream big and work for it all these tokens I have available during training those tokens are vectors on top of which position embedding is added then you construct the attention weights matrix you can calculate the values queries and keys for all these six tokens so I have value vectors of all the six tokens available however if I am currently considering the prediction of the token called big what that indirectly should mean is I should not consider the value vector corresponding to big and work for and it I should only consider one value vector which is the value vector corresponding to big and work 4 and it i should only consider one value vector

12:25which is the value vector corresponding to dream why because if i was if i were doing inference i don't have any future tokens available so because of that reason if you were to somehow magically convert all of these numbers to zeros and only the first number remains which is one what do you think the context vector will become? The context vector will be 1 multiplied by v1. So let's say the value vectors are v1, v2, v3, v4, v5 and v6. They will also be the same vectors which are living in the same dimensional space as that of the query space and key space. So now what will be the context vector if I am predicting the token called big? It will be attention weighted sum of value vectors. What is attention weighted sum of value vector in this case?

13:10One multiplied by V1 plus zero multiplied by V2 plus zero multiplied by V3 plus so on and so on. What does that mean? It means all these five value vectors are being discarded. Why is that? Because the attention weights corresponding to those value vectors are all zeros So effectively whenever when you are trying to predict the next or the context vector corresponding to the next token, which is big Only we want the value vector corresponding to the token available up until that point only that is available Now let's look at the second case. Let's say I'm trying to predict the context vector while doing the next token prediction corresponding to AND. So what all tokens do I have available? I have dream available and I have big available.

13:55So ideally if that was the case, V3, V4, V5 and V6 should not be coming into picture. And here is how. So what is again context vector context vector is attention weighted sum of value vectors so context vector should be 0.48 multiplied by v1 plus 0.52 multiplied by v2 plus 0 multiplied by v3 plus and so on which means this time v3 v4 v5 v6 are being ignored only v1 and v2 is coming into picture. So here this is a very smart mathematical trick. So we are not touching the value vectors. To make the value vectors not exist during the calculation of attention scores, instead of eliminating value vectors, what are we doing? We are eliminating attention scores or attention weights rather. We are making sure that

14:43these five attention weights are zeros. In this case when we are predicting the third token we are making sure that these four attention weights are zeros. In the next case we are ensuring that these three attention weights are zeros. So when attention weights are zeros it's effectively as if the value vectors corresponding to them do not exist because you are multiplying attention weight with the corresponding value vector. So this attention weight, it is the fourth attention weight in that row, which means it will be multiplied with V4 and this attention weight is the fifth attention weight in that row, which will be multiplied by V5 and this is the sixth one in that row, which will be multiplied with V6.

15:18So when these three things are zeros, it is effectively as if you are eliminating V4, V5 and V6 and you only consider v1, v2 and v3. So we are yet to discuss how this attention weights matrix was obtained in the first place, but that we are getting into later soon. But here we are just saying how this is helping. So I hope this part is clear for you. When we are doing attention weighted sum of value vectors, only the value vectors up until that point, whatever is the current token current context vector that is being produced only the value vectors up until that point should be considered and how do you how do you try to accomplish this you accomplish this not by eliminating value vector directly but by converting attention scores or rather attention

15:59weights into zeros okay that is how we do it but that being said the remaining attention weights should sum up to one so this 0.33 plus 0.37 plus 0.3 they add up to one these two numbers they add up to one why because when when we are saying attention weighted sum of value vectors whichever value vectors are being considered the weights of weights that we are multiplying those value vectors with they should add up to one so every row should still add up to one even if the upper triangular elements are being converted into zeros okay now let's see how this is being done or what are the different ways in which this can be done that's the part two of today's lecture so we could do one simple thing first we know that to get

16:42attention weights we take softmax over the attention scores attention scores are raw dot products between query and key Okay, so this is let's say query number key number one key number two key three key four this key Query one query two three four. So this is Q 1 multiplied by Q 1 dot product with K 1 this is Q 1 dot product with K 2 Right. So these are raw attention scores, which we convert into Row being summing up to 1 using softmax. So once you take softmax this row will sum up to 1 This row will sum up to 1 and those are called attention score attention weights raw attention scores converted into probability distribution now what we could do is once I have the Attention weights I could

17:26simply convert these numbers into zeros. I could simply say, oh, let the upper triangular elements be zeros. Then do you think these numbers will add up to one? It will not add up to one, right? Because it was adding up to one before we did this. Before we did this upper triangular elements being zero, it was adding up to one. but now since some of those numbers are being eliminated and replaced with zeros the rows will not add up to one so now but we know that the rows should add up to one so what do we do we can do something like we renormalize so we can say the each element in the row is that element so let's say this element we call as q1 dot product with k1, we divide that with sum of whatever is that row sum.

18:09So let's say the sum of this row is 0.9, that means this element will be divided by 0.9. And same thing we can do here. So if you want to convert this row into something with sums to 1, we can say, okay, every element will be divided by the row sum, whatever the row sum is, that is a way to normalize. That also will satisfy our criterion that every row has to add up to 1, that condition is satisfied. Upper triangular elements should all be zeros, that condition is also satisfied, right? So are we not happy in this case? Is there any problem here? Like my question is, first we take softmax to get attention weights, attention weights, WTS for weights. And here every rows are already adding up to 1.

18:56Now what we do is, we manually convert the upper triangular elements into zeros, and then remaining elements row wise, we divide by the row sum whatever is the sum of the row which means the rows should again add up to one that is a probability distribution and upper triangular elements are all zeros so we get masking is this okay or what is there any problem with this approach there is one problem with this approach that the problem is that we when we take softmax okay so here we said that we are we have already taken the softmax we are already considering the query dot key dot product between the future tokens that is already done you know how because softmax is basically e raised to xi divided by sigma e raised to xi

19:42okay so if we are doing softmax row wise or let's say we take the softmax of first row what are the softmax numbers e raised to x1 divided by sigma e raised to x1 over the row then this one is e raised to x2 divided by sigma e raised to xi summed over the row so essentially when we are saying that if the denominator is constant right to calculate the softmax of this or this or this or this the denominator is nothing but the sum of exponent of every element in that row correct. So what we are saying is the moment we calculate softmax the future attention scores resulting from the dot product of Q and K, we have already baked into every single attention weights. So the moment you calculate attention weights from raw attention scores using softmax,

20:32some information from the future is already baked in. Why is that? Because the denominator of softmax calculation sigma e raised to xi, and what is sigma e raised to xi by the way? It is sigma e raised to query dot key transpose, right? So when we say sigma e raised to xi is the denominator, that number is nothing but e raised to attention dot product, the dot product between query and key. So here we may take the dot product of query and key for future tokens. So query of the current token taken as a dot product with the key of the future token, which means we are indirectly allowing the attention scores to have a look into the future. Because if future token is not generated yet, how come you have the key of that token available

21:19to you? Because if you are working in row number two, let's say your query dot key transpose will be Q2 dotwithk1 transpose. Then the next number will be q2.product with k2 transpose. q2.product with k3 transpose. Right, this is what you are doing. But if you are trying to produce the, you know, context vector corresponding to let's say Q2. How come you have K3 available and how come you have K4 available? Because you don't have that token available yet, right? Unless you produce that token, how come you have this I'm talking about in the inference. In the inference, you don't have K3 and K4 available if you are currently in Q2. Which means when you convert the attention scores into into attention

22:05weights using softmax during training you are indirectly injecting the information about the future into those attention weights so if you already take softmax and then zero the future zero the future meaning zero these numbers which are query dot key transpose of the future tokens future tokens meaning if i am in q1 k2 k3 k4 are all future if i am in q2 k3 k4 are future right so this this won't work because uh or i should not say this won't work maybe this will work to some extent but still still here uh we are allowing the dot allowing the attention scores to have a look into the future so that is the reason why we should not take softmax first and then convert the upper triangular elements into zeros that is not

22:50the right approach because the query dot key transpose includes the key of the future tokens which we don't have yet if we were doing it if we were doing inference so we to mimic the behavior during inference during training also during training this is not something that we do so what do we do what is the alternative let's see what's the alternative the alternative is very simply this so let's say these are the raw attention scores okay in one row in one particular row you have let's say three numbers two three five in the previous example each row were four items but now you might be you might be wondering how suddenly it became three because those these are two different examples that's all.

23:26So we are considering one row, in that row let's say these are the attention scores and let's say we are currently at Q1 which means K2 and K3 should be masked. So what do we do? The second item in that row will be assigned to be minus infinity. Third item also you will assign to be minus infinity. Now what are the numbers on top of which you will apply softmax? You will apply softmax over these numbers. So before you create an upper triangular matrix that are of zeros, you will create an upper triangular matrix that are all minus infinity. So when you take this softmax what will happen is e raised to minus infinity divided by sum of the row right so in this case the sum will be e raised to 2

24:13plus e raised to minus infinity plus e raised to minus infinity what is e raised to minus infinity e raised to minus infinity is 0 right because 1 by e raised to infinity infinity? e raised to minus infinity is 0, right? Because 1 by e raised to infinity is a very big number. So e raised to minus infinity is 0. So what will happen is the softmax score corresponding to this element will become 0. The softmax score corresponding to this element will become 0. The softmax score corresponding to this guy will become 1 because that will be e raised to 2 divided by e raised to 2 plus e raised to minus infinity plus e raised to minus infinity. So this is 0, this is 0 and whenever you have e raised to minus infinity in the numerator

24:50for those elements the attention scores will become 0s, rest of the elements will sum up to 1. So this is essentially what you do. So now how does it look like from a triangle perspective? From a triangle perspective, from an upper triangle perspective it will look something like this. I will convert this into minus infinity, this will be minus infinity and this will be minus infinity. Similarly this will be minus infinity, minus infinity and this will be minus infinity. Now after we do this and this is done for what? Attention weights or attention scores. We convert the attention weights matrix or attention scores matrix into this upper triangular minus infinity which one? It's not attention weights. It's attention scores because we have to do this before we take softmax

25:36So I tend so this number whatever is here It will not add up to 1 because we are yet to take softmax Then if we take softmax what will happen is this row will become 1, 0, 0, 0. Similarly, this row might have been 2, 4, something, something. And what will happen is after we take softmax, this may become, let's say, 0.4 something. This may become 0.6. They will add up to 1 and then this will be 0, this will be 0. I'm talking about the row number 2. Similarly, row number 3, in that third row, there will be three elements which are non zeros whichever was minus infinity after we take soft max along the row that minus infinity element will become 0 so this way mathematically also it's very very easy to handle because soft max of minus infinity I mean soft max of a set of numbers

26:20where the current number that you are considering is minus infinity will be zero okay so that is essentially the the mechanism over here let me just write it down so that you are one you're clear once and for all so let's say my num my sequence is the cat sat on the and we have to predict mat so this is not the right this is Q corresponding to the so this is Q1. This is Q2. This is Q3 Q4 and Q5 Similarly K1. So here this will be the key corresponding to the key corresponding to cat key corresponding to SAT K4 K5 So when you are constructing the attentions the context vector corresponding to Q1 Okay the context vector corresponding to Q1, what you have is you have Q1, you should not have K2, K3, K4, K5 available to you.

27:33Because if you are trying to construct context vector corresponding to Q1, you only have one token available up until that point. So ideally, we should mask these elements as minus infinity. And this will be whatever, whatever was the original raw attention score, which is the raw dot product between Q1 and K1 transpose. Similarly, if you are trying to construct context vector corresponding to Q2, you don't have K3, K4, K5 available because you don't have those three tokens available. If you were in inference, of course, during training, you have all the data available. But for the training data should mimic the behavior of inference. Otherwise, during inference, your text generation will not work.

28:15So when you are working with Q2, k3, k4, k5 should not be available. So you mask those columns corresponding to k3, k4 and k5 with minus infinity and this will be whatever whatever was the raw attention scores and then you take softmax and what will happen is after taking softmax so these are raw attention scores after taking softmax i'll just show one row one row here what will happen is this will become 1, the rest of the columns will become 0, 0, 0, 0, because minus infinity when you convert to softmax, this is what will happen. So like this, eventually all the minus infinity elements after softmax will be replaced with 0, so that is how the upper triangular zeros are obtained in this case

29:08are we not looking at the future like when we take softmax the future why is the future not coming into consideration because you could ask the same question right in this case isn't the future coming into this into the picture when we calculate softmax, the future is not coming into the picture in softmax because softmax is nothing but e raised to xi divided by sigma e raised to xi, correct? And look at the sigma e raised to xi in this row, the sigma e raised to xi is e raised to 2 plus e raised to minus infinity plus e raised to minus infinity plus, I am talking about the denominator, okay, e raised to minus infinity plus e raised to minus infinity plus, I am talking about the denominator, okay, e raised to minus infinity plus e raised to minus infinity.

29:48This sigma e raised to xi will look like this for the first row. Look at the contribution from the second, third, fourth and fifth tokens. The contribution from the second, third and fourth and fifth tokens is 0 because e raised to minus infinity is 0, 0 plus 0 plus 0 plus 0. So the contribution from all these four elements will become zero so you are uh not allowing the you know attention scores to have a look into the future so if you are currently working with q1 you won't have access to um attention scores or even value vectors corresponding to token number 2 3 4 and 5 okay so this is how you prevent looking at the future now one last concept I want to discuss before I give you a code walkthrough is the concept of

30:33dropout now let's discuss about why we do dropout how we do dropout and after dropping out some attention weights what exactly do we do with the remaining weights so the idea of dropout is that if there is a chance for overfitting after dropping out some attention weights what exactly do we do with the remaining weights so the idea of dropout is that if there is a chance for overfitting meaning first of all why do we do dropout in the first place in any neural network the reason why we do dropout is that some neurons in a neural network so let's say this is a neural network in this layer there are three neurons in this layer there is one neuron when we do dropout what happens is let's say and the

31:10dropout probability for this layer is um one by three so there is a one by three chance that these elements will get dropped out so let's say in any given instance, maybe one of these elements might get dropped out, let us say. So what will happen is, to calculate the sum over here, which is w1 x1 plus w2 x2 plus w3 x3, the w2 will not be taking part. Only this neuron and this neuron will contribute to producing an output in this instance, when you have dropped out an element. But there is a problem with that, the problem is that when you drop out one out of three elements in a layer, then your average output intensity which is over here may reduce by a factor of one third, right, which means you have to multiply with a certain number,

32:02if so, so basically if your original intensity was x it will reduce by one third meaning it will become 2 2 by 3 x why because you have only two out of three neurons available in producing the output which means if you want to maintain the output magnitude on an average you have to multiply by 3 by 2 okay so you have to have a multiplication factor when you are dropping out certain elements so that the intensity at the output or the magnitude at the output is not being reduced. So in attention weights, the dropout is implemented with the same idea. So if dropout probability P is 0.5, what it means is every element has a 50% chance of being dropped out so after dropping out it may so happen that this item is gone maybe this is gone and then

32:51maybe this is also gone this may happen it's not always exact there is no guarantee that always exactly 50% of the items will be gone there is there is no such guarantee but every element has a 50% likelihood of being dropped out okay now what do you do with the remaining items? Same thing. If 50% of the items are gone, the remaining 50% has to produce an output of same magnitude as what it was before. Because if the output magnitude is reducing, then your loss will be obviously be increasing because the dropout itself should not contribute to loss of tremendous number. So output magnitude on an average should not change too much. So what you do is if half of the items are dropped, remaining half of the items which are roughly remaining items which may remain should be multiplied by a factor of 2.

33:35So the remaining item, this 1 is remaining, right? So 1 will be multiplied by a factor of 2, it will become 2. 0.52 is remaining, right? It will become 1.04. 0.33 is remaining, which will become 0.66. 0.3 is remaining, it will become 1.04 0.33 is remaining which will become 0.66 0.3 is remaining that will become 0.6 and so on so this is the idea of dropout so now these elements will produce on an average an output of similar magnitude as that of what it was doing before the same idea in neural network so let's say this is this is a particular weight, 0.27 is a weight. So what will happen is half of that, there is a 50% likelihood that it may get dropped out. It means half of the times it is kept, half of the times it is dropped out on an average.

34:17So half of the times whenever it is kept, the magnitude is 0.54. Half of the times when it is dropped, the magnitude is corresponding to that is 0.54 half of the times when it is dropped the magnitude is corresponding to that is 0 so whatever is the resultant output that this particular weight is going to contribute to that output on an average its magnitude will remain 0 plus 0.54 divided by 2 so that is basically the idea so so that and by the way the dropout is only done during training during inference you don't do dropout. You use all the available weights. But during training, to prevent the models from learning nuances in the text and kind of rote memorizing something, you randomly switch off certain neurons.

34:57So here the attention weights are not exactly, I don't like to call it as neurons. It's slightly different from the neurons in the neural network but yet they are trainable i mean they are parameters which are resulting from uh your query and key not trainable parameters they are resulting from query and key so when you drop it out at a with a random probability those way those attention weights do not have a chance for rote memorizing something that is supposed to learn from the text that is a that is a basic idea so if the dropout is with a probability of p you have to multiply the remaining elements by 1 by 1 by 1 minus p so if p is 0.5 1 by 1 minus p will be 1 by 0.5 which is 2 if p is 1 by 3 if things are getting dropped out at a probability

35:42of 1 by 3 remaining will be 1 by 1 minus 1 by 3 which is 1 by 2 by 3 which is 3 by 2 so remaining items will be multiplied by a factor of 3 by 2 so this is in one diagram everything summarized that we have looked at since the last i would say three or four lectures you have inputs input tokens x converted into uh query key and value space using the query weights matrix, key weights matrix and value weights matrix, which are denoted as WQ, WK and WV. So initial inputs may live in a two dimensional space after projecting it into a query key value space, the dimension may change and the resulting raw scores from the dot product between queries and keys are called attention scores. Attention scores will not add up to 1 along the row.

36:30Now what do we do with attention scores? Attention scores we can scale first of all by dividing by square root of d. And once we have the scores, once we have the scale scores, you apply upper right triangular minus infinity mask. So all the upper right triangle elements are converted into a minus infinity number. Then you take softmax. So all upper right triangle elements become zeros. Remaining elements will add up to one in every row. And once you have this, you apply dropout. In dropout, you drop out at a certain probability p, which means there is a p percentage or p times 100% chance for every element to be dropped out. The remaining elements which survive without being dropped out will be multiplied by 1 by 1 minus p. And that will give you an output magnitude

37:16which is comparable to the magnitude which you had before you did the dropout. And this is ultimately what happens to attention weights and then these attention weights are used for computing your context vectors. So ultimately what is happening is your x which is an input is getting into context vector which you call as z. So z is the space in which context vectors are living thus the dimensionality of that space may or may not be different. But ultimately these context vectors when it comes out of the multi-head attention mechanism which we are yet to discuss we will discuss in the next lecture ultimately the dimensionality is similar to that of the dimensions or same as that of the dimensionality of the input space in multi-head attention the

37:59context vectors resulting from every single attention head will have a different dimensionality as that of the input and they are concatenated and they are mixed which we will discuss in the next lecture but this is essentially what happens so now with this we are good to discuss in code how this whole implementation looks like it's actually very very uh simple so first let's say i have six vectors dream big and work for it. So these are the six input vectors or input embeddings and these embeddings are all living in three-dimensional space. And now let's say the query key value space is living in a, query key vectors are value, value vectors are living in a two-dimensional space. So you are essentially projecting from

38:42a three-dimensional space which is the input embedding space into a two dimensional space why do i say this is three dimensional because every row has three items so the dimension number one two and three then you can define self-attention class this we have not discussed how this class is defined strictly in the last lecture but if you look at the construction of the class it's very simple you have uh and i won't be going through the the details of the code right now that we can we have plenty of time to discuss in the future lectures but basically nn.linear means you have a linear projection layer with learnable parameters where input has the in number of nodes output has the out number of nodes and that will form the query weights matrix.

39:26Similarly, you have the key weights matrix and value weights matrix. Then queries are nothing but query weights matrix multiplied by x, same for key and same for value. So here essentially we are passing the x into w.query function but essentially what's happening is this input is being multiplied by a transformation matrix of dimensionality d in comma d out so resulting numbers are queries keys and values attention scores are queries dot product with keys transpose attention weights are basically this converted into softmax format so and after dividing by square root of d so here keys dot shape raised to 0.5 is the square root of d square root of d and by the way what is d d is that the dimensionality of the input embedding space no d is the dimensionality

40:18of the query key value space or we can simply say it's the dimensionality of the key space so dimensionality of the key space raised to square root of dimensionality of the space or dimensionality of the space raised to 0.5. So those are the attention weights. And finally, attention weighted sum of value vectors is the context vectors. So context vector is nothing but attention weights multiplied by values. So now you can simply run this block of code and bring the resulting context vectors, resulting context vectors in this particular example lives in a two dimensional space, not in a three dimensional space. So you had six input embeddings or input vectors and you have six resulting context vector, okay?

41:02So I hope this part is, everything until this part is very clear for you now let's just print the attention weights which are unmasked so unmask meaning you have attention uh weights after taking one by square root of d and after taking softmax so these are these are the attention weights and every row will add up to one so you can see that this row will sum up to one this row will sum up to one this row will sum up to one now we discussed two types of ways to implement uh masking we can directly take these attention weights and master upper triangular elements by directly converting it into zeros all the upper triangular elements are zeros and row wise remaining elements we can divide by the sum of the row.

41:46That is the wrong approach we said, right? So that wrong approach I'm just implementing here. So here what we are doing is we have these attention weights and attention weights are basically converted into all the upper triangular elements are converted into zeros and remaining elements are divided by sum of individual rows. So this is called as naive masking. You can call it as naive masking. And here the resulting numbers are, of course, this will resemble the mass self-attention. From numbers, you will never be able to say whether this was a result of mass self-attention with minus infinity or without minus infinity kind of technique. This is without minus infinity technique. Here you are directly assigning upper triangular elements to be zeros after taking softmax

42:31and then remaining elements in the row you are converting it into a probability distribution by dividing by the row sum. The second way is the correct way where you first create an upper triangular mask of negative infinity then take softmax a longer row so that this guy will become 1 0 0 0 0 as you can see here the next row will become something such that the first two elements will add up to one remaining elements will be zeros that is what this is same thing same thing goes for the next row fourth row fifth row and so on okay and in this block i am simply calculating the sum of every row so some of the first row is one some of the second row is one some of the third row is one so the row sum should add up to one again very very simple concept so

43:18the reason why i'm not going to i'm going i'm not going through these individual lines of code is because it's very simple my my purpose here is not to teach you how to code this attention mechanism. It is just to give you how it may look like when you are implementing it in Python code roughly. And last part is how dropout is implemented. So dropout is implemented like this. So first, you have a, like this you can call as something like a dropout mask which is applied element wise so if you are doing 50 percentage dropout and each time when I run this you will see that this mask will look a little bit different let me see actually it's not different because I'm passing a seed if I don't have the

43:56seed each time it each time I delete should look different yeah see each time each time it looks different but if you want repeatability randomness with repeat repeatability i can have this seed i can enter some seed number so that every time you run the random numbers are generated using this seed number so the mask will look exactly the same. Now what do you do with the mask? You take the same set of attention weights you got over here. So these are the attention weights you got. Element wise you multiply with this mask, with this dropout mask. So what will happen to the first element? First element was actually, first row was 1, 0, 0, 0, 0 like that right. The first element will become 2, rest row was 1 0 0 0 0 like that right the first element

44:45will become 2 rest of the elements were 0 anyways similarly second row every item will become 0 why is that because the first two first two dropout mask items are 0 which means and remaining items were 0 anyway in the second row third row in third row what will happen is third row has had three elements all the three elements will get multiplied by two so where were the three elements these were the three elements so 0.33 0.36 0.30 so that will become 0.66 0.72 and 0.60 as you can see here um oh i'm not sure if this was implemented once again okay this time the mask changed a little bit but let's let's make sure that the mask is being applied as we expect so or celebrates after dropout is two times two times so every item in the in this row should become twice but my question is why is this guy zero okay

45:54okay I think I understand the problem what we can do is we can we can generate a new variable called dropout mask equal to this okay and then I can simply bring that dropout mask and then weights so if I simply multiply it should produce I think should produce a dropped out attention weight so now let's see so now this is the dropout mask that mask is being applied to our original causal weights which are this maybe i can print the causal weights once here just for your reference so let me just print it okay so these are the original causal weights and this is the dropout mask and if these three elements in the dropout master are all zeros all these this row entirely will become zero as you

47:02can see here similarly this item is zero in drop Similarly, this item is 0 in dropout mask. This item is 0 in dropout mask so this guy will become 0 this guy should become 0 as you can see here and What will happen in let's say first row is the first item which was 1 is getting multiplied by 2 So this should become 2 in the third row all items are being multiplied by 2 So this should become 0.66 0.73 something and 0.60 as you can see here okay so this is what happens when you apply a dropout and that's all so eventually this is we can literally write everything inside one class so this is the causal attention class and in this class we define the w query key value weight matrices. We can also define dropout and finally what comes out of this will be context vectors

47:53which are passed through the which are resulting from the dropped out attention weights matrix and the context vectors here will be looking slightly different from the context vectors which you got earlier without dropping out but on an average producing these kinds of context vectors are better if you are doing training during inference you don't do the dropout because during inference you are not trying to prevent overfitting or anything during inference you are just trying to produce output using the already trained language model or already trained neural network so that is the this code in a brief nutshell so we are good to wrap up today's lecture so let's just see what all things we have looked at so in masked attention you are not applying

48:38mask you are not randomly masking the upper triangular elements after taking softmax you are doing the opposite before taking softmax you apply upper triangular negative infinity then you take softmax so that future keys are never present in the calculation of the current context vectors then rest of the things i think you have already you are already familiar with so if you were to take away three things from today's lecture first first thing is that context vectors are attention weighted sum of value vectors. And when you lose the sum of those value vectors, only the value vectors up until the current token should be available in producing the context vector. Any future value vector should not be considered

49:18into the equation. Similarly, when you calculate the attention weights also, future keys should not come into picture. So, to prevent both of this from happening, you have to somehow convert the upper triangular elements all into zeros. But directly converting upper triangular elements into zeros after you take softmax, it will prevent the value vectors, future value vectors from being present in the context vector calculation, but it will not prevent the future keys from being present in the context vector calculation but it will not prevent the future keys from being present in the context vector calculation. The future key will still be present if you are simply converting upper triangular elements

49:51into zero. So what you have to do is before you take softmax you apply minus infinity to all the upper triangular elements so that future keys are not participating because all the contribution from the future keys will become e raised to minus infinity which will be which will be zero so all the keys in the future will not contribute to calculation of the present attention weights and all the future value vectors will not come into picture because the query dot key transpose dot product which are attention weights resulting in the attention weights matrix will all be zeros for the future values. So we will get a clean attention weighted sum of value vectors where only the current value vectors are or value vectors up until the current token is used for predicting the next context

50:35vector and also those weights in the weights calculation only the keys up until that point is being considered in calculating the weights. So it clean on both sides so this is a brief summary of today's lecture i'll be sharing the code files and the slides with you please go through it the last thing which we discussed in the lecture was about dropout you can randomly drop out elements but then the rest of the elements you have to scale up accordingly that's the only thing that you have to take away from today's lecture so thank you so much we will meet again in the next lecture when we will discuss about about multi-head attention and that will be the i would say one of the last lectures in the first initial module of this series where all the basics

51:13of attention mechanism is covered and we will be fully equipped to move into more complex attention mechanisms thank you so much take care see you in the next lecture bye