Simplified self attention | The attention family series - Lecture 5

0:00Hi, welcome back to a new lecture from the attention family series. In today's lecture for the first time in this series we will be discussing about self-attention. In fact we will be discussing a simplified version of the self-attention in today's lecture and simplified because this will not have any trainable parameters. So we will look at the trainable parameter based attention mechanism in the next lecture, but today let us look at the self-attention in a very simple manner. So, we will discuss about context vectors, we will also discuss about the attention score calculation when tokens are attending to each other etc. But before that let us do a quick recap of the last three lectures what we discussed.

0:39In the lecture number two, we discussed regarding how recurrent neural networks were used in encoder and decoder in neural machine translation for let us say converting English into French translation and there the problem was one context vector was responsible for capturing everything that is there in the input sequence and that was too much pressure on one single context vector and Badano attention which was also called as additive attention fixed this to a great extent by introducing context vectors that vary depending on what token the decoder is decoding at that moment. So if decoder is writing the French equivalent of the English word cat which is chat the attention scores would be different where the attention score the attention will

1:20be paid maximum to the token cat. the attention score between cat uh the hidden state of cat and the hidden decoder state corresponding to chat that will be highest and this was the fix produced by badano badano's attention and that was the first time that the concept of attention mechanism became prevalent and for that reason that paper has almost 45 000 citations now however one interesting thing was badano attention was not a dot product based attention between uh tokens but rather it was something called as additive attention meaning we had a formula that looks something like v dot tan hyperbolic of w1 multiplied by hidden state corresponding to the encoder plus w2 which is weights multiplied by hidden state corresponding to the encoder plus W 2 which is weights multiplied

2:06by hidden state corresponding to decoder. So, because you are doing the weighted sum of the hidden states in the encoder and decoder this is called as the additive attention here is the addition happening then that is passed through a tan hyperbolic activation function and then whatever is resulting the resulting vector is taken as a dot product with this learnable vector v and the resulting score is the attention score, attention score. But an even simpler mechanism is there which was simply taking dot product between the tokens that need to attend to each other. So in the last lecture we discussed about dot product attention and scale dot product between the tokens that need to attend to each other. So, in the last lecture we discussed about dot product attention and scaled dot product

2:47attention and we discussed that the scaling which is dividing by square root of d is very important to make sure that the softmax scores are not becoming saturated because if softmax scores saturate it will cause issues during training because the loss may become very small and for that reason the LLM training will not be very efficient. So this all of these we discussed in the last three lectures. Now in today's lecture we will be purely looking at self-attention mechanism and now today for the first time we are entering this category or this bucket. We have been coming back to this figure again and again in this series basically this figure is showing the different buckets in which the different attention mechanisms

3:27that we will be discussing in this series is belonging to. So, in the last three lectures, we completed this scoring bucket, meaning how the attention scores are calculated. And there were three items, the Badano attention, dot product attention and scaled dot product attention. And in today's lecture lecture when it comes to connectivity bucket what we are discussing is how tokens which tokens are allowed to attend to which other tokens okay so this is where we are at now one thing to note is when we are talking about self-attention mechanism you will come across three terms, query, key and value. Actually, this query, key and value are all produced from the same input token. So X input is converted into query, converted into key and converted into value.

4:15This is how it is done practically. However, for today's lecture, because it's a simplified self-attention, we won't have any trainable weights, which means query, key and value will not exist as three separate things. If you are not familiar with what is query key and value don't worry because that in this series we have not yet discussed what is query what is key and what is value that we will discuss in the next lecture. So in the next lecture you will have very good clarity or very good intuition on what exactly is query key and value but in today's lecture we will look at this whole process from an even simpler perspective. The perspective is that currently there won't be any trainable parameters involved in deciding

4:51the attention score. So this is the upcoming sequence. I would say it may be four different lectures or sometimes I may decide to convert some of these concepts into one single lecture but essentially we have to cover simplified self-attention after that we will look at attention with trainable weights with trainable weights meaning we will have query weights matrix key weights matrix and value weights matrix to convert x which is input into query comma key comma value this we will discuss in the next lecture after that we will discuss the mass self attention or causal attention and after that we will discuss about multi-head attention so this is the sequence that we are going to follow in the next few lectures so today's lecture is divided

5:36into three parts first we will see the difference between badano attention which was actually not the same sentence looking at itself it was actually the french sentence sentence looking at english sentence so it was technically like a cross attention rather than a self-attention but we'll see the difference between badanau's attention and the self-attention we will look at in today's lecture and we will also see how the embedding which is the vector embedding corresponding to every single token how is it converted into context vector. So embedding we will denote by the symbol x and context vector we will denote by the symbol z and we will see in self attention that we will discuss today how this context

6:15vector is calculated. Of course it is calculated using scale dot product, but we will see the details of how exactly the calculation goes. Then we will look at an example sentence dream big and work for it. So this is the sentence that we will look at which has 1, 2, 3, 4, 5, 6 words which translates to roughly 6 tokens but that is an approximation and each word we will try to represent using a three dimensional vector. So we will have dimension x1, x2, x3. So, we will have a three dimensional vector to represent each of these tokens and we will see using those vectors we will assume some numbers. So, starting with a numerical set of vectors we will see how to calculate the attention scores and we will also see how the entire process of going from tokens which are x into attention scores and

7:09then finally into context vector how can this whole thing be represented by one single equation, one single mathematical equation which is basically involving matrix multiplication and softmax calculation. So we will see that and in fact the formula which is in which was introduced in the attention is all you need paper which is softmax of. So, I will just write it down here although I am not expecting you to fully understand if you are not aware of it, but softmax of query times key transpose query dot key transpose divided by square root of d multiplied by value. So, this is the formula which helps us calculate the context vector for given a query key value which was introduced in attention is all you need paper, but in today's lecture you

7:51will see the equivalent of this formula would be softmax of and we do not have query key value in today's lecture everything is the input x. So, x dot x transpose divided by square root of d, square root of d is the dimensionality square root of dimensionality multiplied by x because we do not have query key and value separately or everything is basically the input token x. So, we will see this formula towards the end of the lecture. So, these are the three parts. So, first let us look at the difference between Badano attention and self-attention. So, in badano attention we said we are having French token so I'll call this as f1 and f2 so this is a French token number one and French token number two and I'll put this as even e2

8:35e3 so what is happening in this particular example is an English sentence with three words is converted into a French sentences with two words so there is no guarantee in translation that the number of words in the input English sentence with three words is converted into a French sentences with two words. So there is no guarantee in translation that the number of words in the input and number of words at the output are the same, right? Translation is dependent on the languages. So and because of that, since badano attention is constructed, such that the French tokens are attending to English tokens. So French is just an example. Okay, by the way, please don't take it that French and English are the only languages it can be any languages but and it can even be paraphrasing it

9:09can be any task but essentially what I'm saying is the attention is constructed between what you get at the output which is the output token and what you have at the input which is the input token which means it is not a self-attention a sentence is not attending to itself the output is attending to the input so because of that reason badano attention in a generalized case it produces square non-square matrices as the attention matrix or attention map it will be rectangular okay because these are two different sequences and the number of tokens in each sequence can be different on the other hand in self-attention the same sentence is attending to itself meaning the tokens within one sentence is attending to all the tokens within the

9:51same sentence which means obviously it will be a when you can when you construct the attention weights matrix it will be a square matrix so basically these are attention weights so if you add up these three numbers it will add up to one if you add up these three numbers, it will add up to one. If you add up these three numbers, it will add up to one. So these three numbers will be, let's say 0.6 plus 0.2 plus 0.2. They will add up to one. This may be 0.7 plus 0.15 plus 0.15. They will add up to one. Similarly, here each row will add up to one. These are all attention weights corresponding to one particular token. Let's say I call this as Q1, attending to tokens in the same sentence. I will name this as K1, K2, K3, K4, etc.

10:32And it will be square matrix because one sequence is attending to itself. So, this is one of the major differences between Badano attention and self-attention. We will also discuss in some of the future lectures the concept of cross-attention. We will also discuss in some of the future lectures the concept of cross attention and at that time you will see that in cross attention there is no guarantee that your attention weights matrix will be a square matrix, though it could be again a rectangular matrix. So it depends on whether we are talking about a sentence attending to itself or tokens attending to themselves or tokens attending to a sequence of another set of tokens. So just look at this idea. So if we have the sentence, the cat drank the milk because it was thirsty. This is the first sentence. I will denote this as sentence sequence one or sentence

11:17one. This is sequence two. So here when we say it, here it was thirsty means cat was thirsty. So here the meaning of it is cat. However, look at another sentence, the cat drank the milk because it was sweet. Here actually it does not mean cat, it means milk because milk is what is sweet, cat is not sweet. So, but ultimately if you look at the tokens, it is the exact same token. But we can't have a fixed embedding for the token. If we had a, let's say these tokens are represented using three dimensional vector, x1, x2, x3. We can't have the same vector representing it in both of these cases because clearly the meaning of it in sentence one and sentence two are different. So we have to have a mechanism to represent this or change if there is

12:14a base embedding for the token it then base embedding somehow we should modify based on the available context. So here what is available context this entire sentence is the available context based on this available context we should be able to somehow modify the base meaning or the basic embedding of it basic embedding is by default the term the token it may have a vector embedding that vector embedding somehow we should be able to modify depending on what is the context and the context is clearly different in the sentence 1 and sentence 2 and because of that the resulting vector representation of it should also be different. So what we have to do is the embeddings which are represented using x, so in this case I

12:58am taking another sentence here dream big and work for it. In this sentence x of 2 means this item item this is x of 2 or i can simply call it as x with underscore 2 does not matter it depends on what what is our convention x 2 is the actual vector embedding but this vector embedding we can't take directly because this vector is not you know this token or this word is not standing in isolation this is available within a context so because of the context because of the available context we have to modify the embedding into something called as context vector so self-attention mechanism is the art of how to convert this X into Z how to convert the the default input embedding into a context vector that is the whole question of

13:52what the self-attention is going to so now let us look at some numerical examples okay so I am going to take the same sentence dream big and work for it. So dream big and work for it Here I have taken a random Three dimensional vector random set of three dimensional vectors as the corresponding vector. So this is the vector corresponding to dream this is the one corresponding to big and Work for it. So totally we have six vectors, each vector is living in three-dimensional space. Obviously, needless to say in large language models, the embedding space is much bigger, it can be 768 dimensions or 1024 dimensions depending on the exact architecture. Now, what will happen is if we draw these six vectors in three dimensional space, which

14:46in fact we are going to do in some time, there is a simple Python script that I have written, which I will walk you through. I will also share the link to this Python script. You will see that when you plot these vectors, some of these vectors are pointing to a certain direction of space, like dream, big and work may be pointing to one space in the semantic space or one area in the semantic space and these two tokens and and four which are kind of filler words they are pointing in another direction and it is pointing in another direction so this may this may happen this we had discussed in the last lecture also so you may if the sentence has words which are like apple, mango, oranges, etc. they all may point into a certain direction

15:30of space which may correspond to fruits. So this is a semantic space corresponding to fruits and if there are bunch of animals which are mentioned in the sentence like lion, zebra, etc. they may those tokens the embeddings corresponding to those tokens may point into a different direction different region in the semantic space corresponding to animals ok. So, vectors carry meaning the direction as well as magnitude there is some significance associated with it. Now, let us look at the self-attention scores, how the self-attention scores can be calculated when the token big and because we said self-attention, it is like you pick one token from the sentence and you will let that token attend to all the tokens within the same sentence.

16:17Meaning, we will take big as the input token which is x of 2 or x 2 and we are saying what is the what is what are the attention scores if x is the token which is attending to every other token so x x meaning x of 2 which is big so big attending to dream big attending to itself then big attending to and big attending to work and like that so totally when when the token big is attending to itself and to all the other tokens in the same sequence, first of all, how many attention scores will be there? The total number of attention scores should be equal to total number of tokens in the sequence. So in this particular sequence, we have 1, 2, 3, 4, 5, 6. We have totally 6 tokens and now right now big is attending to all the tokens.

17:10And as a result we have currently 6 attention scores. The best way to denote this attention score is probably using some notation like omega or something. So we can call this as omega 2, 1. This is omega 2, 2. this is omega 2 2 this is omega 2 3 so in my notation omega this 2 underscore 2 denotes that the fact that i am taking big which is the token number 2 as the token as the query token which is now attending to all the rest of the tokens so now omega 2 1 omega 2 2 omega 2, omega 24, omega 25 and omega 26. Those are the raw attention scores. Now you may have a question, how come the value of the and how are these scores by the way calculated? It's simple dot product. You are taking the vector corresponding to big and you are taking the vectors corresponding

18:04to all these tokens and you are performing dot product. So, the dot and the result of the dot product is a scalar right scalar number. So, you might have a question when you perform dot product how come the dot product between big and itself. So, the dot product between big and big how come that is not the highest because they are pointing in the exact same direction, right, because big as a vector is pointing in the same direction as itself, which means the value of that should be highest, no, but there is a small thing to note, the thing is that, yes, that value could be highest, but there is also magnitude to these vectors, so to give an example, this is a very simple example, let us say this is the vector corresponding to big and let us say this is the

18:49vector corresponding to maybe work. So this is the vector corresponding to work and this is the vector corresponding to big. Now one thing to notice when we are creating embedding space the magnitude of all the vectors are not equal to one the magnitudes can vary so if big is a vector which is pointing to itself then here 0.91 is basically the dot product like something like x2 dot x2 but you can imagine that when you are taking the dot product between work and big you are essentially taking a projection of work onto this line, right? So you are essentially saying, and what is work? Work is x4. So x4 so x4 so when you do the dot product of x4 dot x2 you get something like x4 x2 cos theta where cos theta is the this theta is the angle between the

20:00some reason I am not able to select one second so this theta is the angle between the two vectors and you can already imagine that if theta is a small number so if this angle is a small number then x4 dot x2 can actually be bigger than x2 dot x2 and that is what you are seeing here so here the dot product between x2 and x4 is 1.04 which is resulting in omega 24 whereas the dot product between x2 with itself is only 0.91. So, this can happen because the vectors are having different magnitude. So, just pay attention to that. Now, what is happening here is this dot product it is rewarding the length of the vector as well as direction right. So, this you already saw. So, in this case when big when the token big is attending to all the tokens in the same

20:58sequence the attention score is highest between big and work ok. So, this is just a distribution of the attention score is highest between big and work. So this is just a distribution of the attention scores. But attention scores, the raw attention scores, we do not use as such. We obviously convert the attention scores using softmax into attention weights. So there is one more question that we could ask ourselves. If our goal is to convert the scores into something which will add up to 1 and let us say the scores are something like 1, 2, 3, 6. Let us say these are the raw attention scores. Why can't we divide each of these scores by the sum? So sum is 6 plus 3, 9, 9 plus 2, 11, 11 plus 1, 12. So why can't we divide each of these numbers by 12 and that way we get a probability distribution

21:47which will add up to one right so here we get something like this this distribution will add up to one the problem with this is the dot product when you take dot product between vectors if the angle between them is let's say greater than 90 degree the dot product itself can be a negative number so what will happen is you will end up getting probability scores which are like negative numbers or rather attention scores which are like negative numbers you don't you don't want that you want to say either these tokens attend to each other or you say you these tokens do not attend to each other at all if they don't attend to each other the score should be very close to zero if they if they if they attend to each other very well

22:22the score could be very close to one right that is what you are trying to say with the attention weights when you convert the attention scores into attention weights that's what you want to do so softmax actually takes care of that issue even if the attention scores are negative or any sort of numbers softmax will ensure that the resulting numbers are one positive and second it will also ensure that the sum will add up to one so here these are the resulting softmax numbers and you can also already see that the dominant number here was six and here again you can see the fourth number the fourth item in the sequence is the dominant one i mean here i'm taking a different example okay so in the previous

23:01example there were six attention scores here there are only four attention scores so please don't get confused by seeing now suddenly four attention scores and over what happened to the remaining uh two attention scores no this is a completely different example which i took to tell you why we are doing softmax rather than just you know dividing by the sum of these numbers so we take softmax due to the reason that we want to convert these numbers into a probability distribution, the sum should be equal to 1 and every number should be positive. Now, the scores were denoted by omega, right? Omega 1, 1, omega 1, 2 or in this case omega 2, 1, omega 2, 2, omega 2, 3, etc. So, these are the attention scores which will be negative also

23:44depending on the dot product value and they will not add up to one and attention scores Omega is converted into attention weights alpha and alpha is basically the probability distribution so here what we are essentially looking at is alpha 2 1 this is alpha 2 2 this is alpha 2 3 and so on so what does this mean this is the attention weight where the token number 2 is attending to token number 1 this is the attention weight when the token number 2 is attending to token number 2 itself basically the token attending to itself this is the attention weight when the token number 2 is attending to token number 3 and so on. So these alphas will add up to 1, omegas will not add up to 1. Now let us proceed ahead and have a look at how the context vector will look like.

24:38Okay, now what are we going to do with these alphas? Because we got the attention weights fine but what's our ultimate goal our ultimate goal is not to simply have bunch of attention weights our goal is to use these attention weights to produce context vectors so our goal right now is to convert x of 2 into z of 2 how do we do that how do we convert x of 2 into z of 2. How do we do that? How do we convert X of 2 into Z of 2? What we do is we have 6 attention weights, right? So, Z of 2 22 multiplied by x of 2 plus alpha sorry this is not 32 this is 23 alpha 23 multiplied by x of 3 so on until x of 6 and how did we get alpha? We got alpha to 1 or basically alpha equal to, let me just delete this, alpha is equal to softmax of, how did we get alpha? Alpha was the softmax of

25:59dot product between x with itself, x dot, in fact in terms of matrix notation we have to say x dot x transpose. One step I have not mentioned here is when you get the raw attention scores these omegas you divide it by square root of d as we discussed in the last lecture to make sure that the softmax scores which we get when we convert attention scores into attention weights should not saturate. So this divided by square root of d is also there. So this is how we got alphas and now those alphas are being multiplied by x again. So in terms of matrix notation we can simply say context vector z in our current case is equal to softmax of x dot x transpose divided by square root of d what is square root of d square root of dimensionality of the embedding space in this case it will be

26:55square root of 3 because it's a three dimensional space multiplied with x so that is essentially what is happening and if you open the attention is all you need paper and if you look at the formula, the formula there will be pretty similar to this except x will be replaced by query key value. I will quickly show you the formula, but in detail we will discuss this in the next lecture. If you are already familiar with all of this by the way, you can feel free to skip this lecture. I am sure some of you are already familiar with this and you might find this very elementary if that's the case no no worries like you don't have to follow this lecture necessarily however if you want to use it as a refresher feel free to do that so this is the

27:34attention of qkv essentially what is this attention of qkv it is the context vectors so context vectors is equal to softmax of query dot key transpose divided by square root of d multiplied by v. In our case, query key and value everything is the input x itself because we are implementing a simplified self-attention. So, our formula will look something like this softmax of x dot x transpose divided by square root of d multiplied by x. Okay, so how did we get z of 2? Z of 2 is basically X1 multiplied by alpha 2 1 plus X2 multiplied by alpha 2 2 this formula or alpha 2 1 multiplied by X1 plus alpha 2 2 multiplied by X2 plus alpha 2 3 multiplied by X3 and so on. So that is our context vector Z of 2. What if we wanted to calculate

28:22context vector corresponding to Z1 basically let's say token corresponding to dream is called x1 what if i wanted to convert x1 to z1 what will we do first we will calculate the omegas basically omega 1 1 omega 1 2 etc up to omega 1 6 we will calculate then we will divide those omegas by square root of d which is the normal scaling. Then we will take softmax to convert them into omega alpha 1 1 alpha 1 2 etc up to alpha 1 6. Now we are ready to calculate z 1. What is z 1? z 1 is equal to alpha 1 1 multiplied by x 1 plus alpha 1 2 multiplied by x 2 plus etcetera up to alpha 1 6 multiplied by x 6 that is the final context vector corresponding to token number 1. Like this you can calculate any context vector and did we discuss any trainable parameter

29:23in this whole case no, the only question you may have is how did we get the input embedding in the first place? How did we get x1? How did we get x2? How did we get x6? That question you may have. That you can imagine we have an already defined. Let us say you imagine that we already have a well-defined library. Like a word-to-vec library or basically a set of embedding matrix. Which tells us if this is the token, this is vector this is the token this is the vector the way it is created in the first place is through is through training a very big neural network but that we are not going to discuss right now it is a it is a separate discussion altogether but once you have the base word embeddings to calculate the context vectors it is all

30:01about calculating the attention scores like this. So here to calculate alpha 1 1, alpha 1 2, alpha 1 6 etc we did not use any trainable parameter we use the input embeddings directly and using the input embeddings themselves we generated these alpha values. But in reality when in the real attention weight or attention score calculation this is not what happens ok. Now the last part of the lecture, let's look at this whole thing overall. So first, look at this dimensionality. Okay. X is 6 by 3. Why is X 6 by 3? Why is it 6 by 3? X is 6 by 3 because we have six words, dream, big, and work for it. We have six words and each word is living in a three-dimensional space so this is word number one dream big work and work for it and this is the dimension one

30:53dimension two dimension three versus six by three so this is the input embedding now input embedding multiplied with the transpose of itself so three by six is basically x transpose what do we get we are essentially getting dot product we how many dot products will be there the resulting dot product matrix will be 6 by 6 so when you are multiplying a 6 by 3 matrix with a 3 by 6 matrix you will get a 6 by 6 matrix so this is equivalent to taking dot products and here what you have is attention scores i mean if you take softmax if you divide by square root of d and then take softmax it will be attention weights, otherwise it will be attention scores. So here we have already mentioned that we are taking softmax row wise.

31:34So each row will add up to one and each row is the attention weight corresponding to that particular token so the first row is basically a set of attention weights corresponding to token x1 second row is the attention weights where we are considering x2 as the token which is attending to all other tokens and so on so totally how many attention weights do we have we have 36 attention weights okay and one more question i have here is are the attention weights symmetric meaning if i transpose this matrix do you think the attention weights will remain the same so my question is is alpha 1 alpha 1 2 is equal to alpha 2 1, is this true? That is my first question. Second question is, is omega 1 2 is equal to omega 2 1?

32:36Omega is by the way attention scores, alpha is attention rates. We can already see that this is true. Omega 1 2 is equal to omega 2 1 by the way omega is true, omega 1 2 is equal to omega 2 1 y, the way omega is calculated omega 1 2 is nothing but x 1 when we take dot product of x 1 with x 2 and then if we do divide by square root of d. So, x 1 dot product with x 2 divided by square root of d is same as x 2 dot product with x 1 divided by square root of d. So, omega 1 2 is same as omega 2 1. Is it same for alpha? Do you think alpha 1 2 is equal to alpha 2 1? Alpha 1 2 need not necessarily be equal to alpha 2 1 because to calculate alpha you are not just dependent on omega 1 2 because to go from omega to alpha you are taking softmax so which means you are dependent on

33:25other omega values also to calculate alpha so there is no guarantee that alpha 1 2 will be exactly equal to alpha 2 1 so just remember that in general this is not true okay the attention weights alpha 1 2 is not necessarily same as the attention weight alpha 2 1 Okay, so once you have the attention weights you multiply that with the matrix again 6 by 3 matrix Which is the same 6 by 3 matrix which is over here. So this is the input embedding, but if this was Attention mechanism with trainable parameters instead of X this would have been the value vector value Matrix V but in this case currently or value when i say value matrix i mean the collection of value vectors but here it's the collection of input embeddings themselves so this is input embedding number one like that

34:15we have six input embedding one two three four five six each of them are three dimensional vectors so you multiply this six by six attention weights matrix with this six by three uh input embedding x to get six by three uh context vector so what what do i mean by six by three context vector the con the six context vectors each living in three dimensional space so this is z1 which is the context vector corresponding to first token this is z2 like that this is z6 so this is how you convert uh you know x1 etc up to x6 into corresponding context vectors z1 z2 z3 etc up to z6 and these context vectors play a very important role in large language models in the next token prediction okay so moving on this is an actual numerical example.

35:05This in fact we will see soon. We can go through this code file once and we will run the different cells and we will see. But basically the attention weights, one particular row if you look at it, it will look like this and the numbers will add up to one. This we will easily see. And I have one question or one thought process for you x2 is the original input embedding corresponding to big right and z2 is the context vector corresponding to the token big x2 and z2 need not be pointing in the exact exact same direction z2 is a modification that we are bringing to x2 using the attention weights so you might see that z2 will change its direction compared to x2 it may point in slightly different direction depending on how much the how much information

35:53the rest of the context is bringing in in the form of attention weights. So, this also we can plot our vectors in 3D and we will see this now let us move on to pytorch yeah this is one last point in today's lecture we are having query equal to key equal to value equal to x because we are not having any weights matrix or we are not having any transformation matrix which converts x into query space x also into key space and x into value space we are not doing that and again when i keep saying query key value if you are not familiar with that term with those terms don't worry we'll be discussing about it in detail in the next lecture so now let us go through the google collab notebook once it's very simple these are the same words dream

36:34big and work for it and we have six vectors total this is the vector number one this is a vector number two vector number three each of them are three dimensional vectors corresponding to each of these tokens so just run this and the shape is six by three six tokens and each of them in three dimensions and here in this block i'm just printing the magnitude of those tokens i'm not going to go through the code line by line because it's a fairly straightforward code okay but essentially i just wanted to print the magnitude of these tokens to show you that every tokens are not of same it's not unit dimension uh unit magnitude the magnitudes are different so clearly here you can see that the token work has a magnitude greater than one

37:14many other tokens some of these tokens are very small uh this is the dream is the token with the with the lowest magnitude now what we can do is we can simply make a basic 3d plot of all these six tokens in the three-dimensional space so in the three-dimensional space you can see that work big and dream these three tokens are pointing into some common uh direction it seems like similarly for and and they are pointing in another direction and it it is pointing in a completely different direction so when we take dot product i assume that dot product between it and the rest of the tokens will be a bit small um whereas dot product between four and four and and will be big dot product between big and work will also be big

37:56and we will also try to convert this to a big this vector which one is the big vector this one this is the vector corresponding to big we will see what happens to this vector once this is converted into a context vector okay let us go down now we are calculating attention score so attention scores are inputs multiplied with the query so what is actually inputs in In our case, we are interested in calculating the attention scores where big is the query and rest of the tokens are all the keys, right? So query equal to inputs of one, meaning from the inputs, the item at the index number one, index number one meaning second item. We are picking what is the second item this one inputs of one is this row this is the inputs variable

38:51and inputs of one is this particular row so now this is the query and so what we are doing is we are calculating attention scores by taking the dot product between inputs and the query inputs is all the six Embeddings and query is just the embedding corresponding to big and this results in six attention scores The as I mentioned Omega 2 1 Omega 2 2 Omega 2 3 etc So the dot product between big and dream is 0.80 1805. This is Omega 2 1 this is Omega 2 2 this is omega 2 2 this is omega 2 3 this is omega 2 4 and so on now these are six attention scores now we can convert in a naive way the attention scores into we can try to convert it into attention weights by dividing by the sum of those scores otherwise the better way to do that is to run softmax so here we will be using

39:46softmax to convert attention scores into attention weights so after converting scores into weights these are the numbers and obviously whichever is dominant so here we can see that the dominant attention score is 1.037 basically the dot product between big and work so even after say taking softmax that will remain that that will be the case so after taking softmax which one between big and work this will be the dominant one see 0.2059 so this is the dominant one and there some will ideally they will add up to exactly one here it is slightly above one because the precision we are not taking into consideration that these numbers have certain floating point precision. Then ok, so this is the final softmax attention weights. Now

40:34what do we do? We use these attention weights and we multiply with the inputs. So, this is the step where we are doing softmax of x dot x transpose divided by square root of d multiplied by x to get the context vector. So, the final resulting context vector z is this one 0.5182, 0.3431, 0.5794 and this is z of 2, z of 2 because this is the context vector corresponding to big and in fact we can also calculate context vector corresponding to all the vectors. So, attention scores will be 6 by 6 matrix which is inputs multiplied by the transpose of inputs. So here we are essentially taking the dot product of all the inputs with itself. And this will be a 6 by 6 matrix as you can see here. And then attention weights will be calculated by taking softmax.

41:21Are we taking softmax row wise or column wise? We are taking softmax row wise. column wise we are taking softmax row wise every row should add up to one every column we need not add up to one every row should add up to one so dimension equal to minus one denotes that we will be taking softmax along the row and here we are simply printing the attention weights so these numbers will add up to 1, these numbers will add up to 1 and so on. And then finally we can calculate the context vectors which is attention weights multiplied by the input embeddings and these are the six context vectors, context vector number 1, context vector number 2, etc. We can plot all the context vectors, but it is better if we plot the same embeddings which we had before and plot one of the context vectors. So here

42:08what I am plotting is I am plotting the same embeddings as before except one item, one additional context vector we are plotting here z of 2. So z of 2 is the context vector corresponding to big, this brown color arrow. This green color arrow is the original input embedding corresponding to big and you can see that the original input embedding has been modified a little bit here based on the available context. If the available context was something else, then the way in which big would have been modified would have been different, right? So this is what I had discussed in the early beginning part of the lecture. In one of the initial slides, I had mentioned that mentioned that you know the cat drank the milk example let me I don't you know I don't

42:51remember the exact sentence let me go back this was the exact sentence the cat drank the milk because it was thirsty versus the cat drank the milk because it was thirsty versus the cat drank the milk because it was sweet. So the input embedding original input embedding of it would be the same but when it is converted so x of it will be the same in both case but what we are doing is we are converting x of it into z of it and do you think z of it will be same in both of these cases? No, Z of it will not be the same in both cases because the available context for it in sentence number 1 is different from available context for it in sentence number 2. So, even if X of the X of it which is input and running of it in both cases are the same, the resulting context vector corresponding

43:44to it will not be the same, they will be different and the same, the resulting context vector corresponding to it will not be the same, they will be different and that is how this whole context vector calculation is adding tremendous value into our whole calculation that we have been discussing so far. Okay, so what we have discussed so far is just a simple attention mechanism without any trainable parameters, so z equal to softmax of x times x transpose divided by square root of d is also there i am not mentioning it multiplied by x okay then we have discussed all of this uh that the recap of this lecture this is fine in the next lecture what we are going to do is we will look at how we can convert the input embeddings into query, key and values.

44:27So the same input embedding, basically X will be converted into Q, X will also be converted into K and X will also be converted into V using WQ, WK and WV respectively. So in the next lecture, we will discuss the intuition behind why we are doing this what exactly is key what exactly is query and what exactly is value those points we will discuss in the next lecture and i think now you are slowly becoming better and better at having a good intuition about the attention mechanism and by the way this is the most simplest attention mechanism mechanisms uh we are we are moving slowly to more complex attention mechanisms in the future lectures. But whatever you saw today is something that will lay the foundation for everything else that is yet to come in this lecture series.

45:18So I think the Google Colab notebook also we have covered fully. I'll share the link to the slides and Google Colab notebook with you. I also have a basic attention, simple attention lab where you can interact with different tokens and see how the attention, raw attention scores look like and how the attention weights look like and how the, you know, each individual row here looks like. You can also try dividing by the sum versus dividing by versus converting into softmax just a basic exercise this we already discussed that's the reason why i did not go through this lab but in any case i'll be sharing the link to that lab if you want to just interact with it and play with it so thank you so much we will wind up today's lecture here see you again in the next

45:59lecture and take care until then bye