The attention family · Vizuara · Transcript & summary by WhipScribe
A clear map of where position information enters attention, with the arithmetic behind each method worked through by hand.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“Clearly the order in which tokens are arranged in a language matters.” Sreedath Panat · at 6:54 — open in the transcript →
“The space, the embedding space is very, very sparse.” Sreedath Panat · at 14:33 — open in the transcript →
Lecture 9 of Vizuara's The attention family turns to position, the piece every attention variant needs and none supplies on its own. Dr. Sreedath Panat starts from a pair of sentences built from the same four words, all dogs are animals and all animals are dogs. Without position information each word has the same embedding in both, so every query-key score comes out identical and the model cannot tell the sentences apart. He also answers a common worry: adding a position vector does not push a word into another word's meaning, because roughly 52,000 tokens in 768 dimensions leave the embedding space almost empty.
The absolute methods add a vector to the input embedding. GPT-2 and BERT learn a table with one row per slot, 1,024 rows by 768 columns in GPT-2, which leaves nothing for positions past the context window. The sinusoidal scheme from the 2017 Attention Is All You Need paper replaces the table with a sine and cosine formula. Fast-cycling low dimensions repeat often, but slow high dimensions almost never do, so every position still gets a unique vector at any length.
The relative methods care about the distance between query and key, not where the pair sits. Shaw and colleagues, with Ashish Vaswani among the authors, add a learned vector to the key, and optionally the value, picked by distance and clipped at plus or minus two, so five vectors cover any context length. T5 instead adds one learned number to the query-key score, chosen from 32 log-spaced distance buckets per head. A tiny GPT-style model trained on TinyStories in Google Colab writes noticeably less garbled stories once position is added. Rotary embeddings and ALiBi are left for Lecture 10.
In Vizuara's Lecture 9, Sreedath Panat uses two sentences made of the same four tokens: 'all dogs are animals' and 'all animals are dogs'. Without position information, 'dogs' and 'animals' get the same embeddings in both sentences, so they project to the same queries and keys and the dot product between them is the same number, even though the word order, and the meaning, differ. Attention on its own ignores order; position information has to be injected so the two scores come out different.
Sreedath Panat of Vizuara argues it almost never does, because the embedding space is extremely sparse. GPT-2's roughly 52,000 tokens live in 768 dimensions, while filling that space even coarsely, at 10 steps per axis, would take 10 to the power 768 vectors. A position vector can be as large as the word vector, yet the sum points somewhere no other word does, so 'cat' plus position stays closest to 'cat', not 'dog'.
Sreedath Panat explains that GPT-2 keeps two lookup tables learned during training: a token table of about 50,257 rows by 768 columns and a position table of 1,024 rows by 768 columns. The position table has 1,024 rows, not 50,257, because GPT-2 processes at most 1,024 tokens at once. A token's row from the first table and its position's row from the second are added element by element. The weakness is that a model stretched to an 1,100-token window has no learned vector for slots 1,024 to 1,099.
Vizuara's Lecture 9 walks through the formula from the 2017 Attention Is All You Need paper: even dimensions take the sine, and odd dimensions the cosine, of the position divided by 10,000 raised to 2i over d. Each sine-cosine pair shares one frequency, and frequencies drop as dimensions rise. Sreedath Panat works it by hand for position 2 in 8 dimensions: dimension 0 is sin(2), about 0.91, and dimension 1 is cos(2), about -0.42. Because it is a formula, not a table, it gives a vector for any position, however long the context.
Sreedath Panat of Vizuara explains that only the low dimensions repeat quickly. In his example, dimension 0 repeats about every 6.3 positions and dimension 2 about every 62, while the highest dimensions change so slowly they never repeat inside a realistic context. Two positions may share values in a few fast dimensions, but the whole vector still differs, and the chance of a full repeat shrinks as dimensions grow. He shows it with a heatmap of 32 dimensions across 48 positions.
In Vizuara's Lecture 9, Sreedath Panat covers Self-Attention with Relative Position Representations by Shaw and colleagues, Ashish Vaswani among them. A learned vector chosen by the distance between query and key is added to the key, and optionally to the value, never the query, since the query is the reference point. Distances are clipped at plus or minus two, so his example needs only five learned vectors, for -2, -1, 0, +1 and +2, whatever the context length. The idea is that nearby words matter more for meaning than absolute place in the window.
Sreedath Panat explains that the 2019 T5 paper, named for Text-to-Text Transfer Transformer, adds a learned scalar to the query-key dot product instead of adding a vector anywhere. Each distance maps to one of 32 buckets: distances up to 7 get their own bucket, wider ones share log-spaced buckets, and anything 91 or more away falls in the last. Each attention head learns its own 32 numbers, so 10 heads mean 320. In his example, a score of 0.9 plus a bias of -0.8 becomes 0.1.
Sreedath Panat closes Vizuara's Lecture 9 with a tiny GPT-style model trained on the TinyStories dataset in Google Colab, once with and once without position embeddings. Test loss is lower with position information, though the gap is smaller than expected, since the network learns some things without it. The generated stories differ more: without position the output is close to a list of words, while with position it shows some sense, such as a girl deciding to go to the park, though still ungrammatical after very little training.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Hi, welcome back to a new lecture from the Attention Family Lecture Series. Today's lecture is for the first time in this series going to talk about position embedding. I was not sure at the start of the series whether we should include dedicated lecture about position embedding or not, but I decided to include this because no matter what attention mechanism we are going to discuss in the rest of the series position embedding is an inherent and important part of it so i thought it was very relevant for you to understand how injecting position information into the either in the input embedding or into the key vector or into the dot product between query and key or wherever else the position embedding in position
0:42information is being embedded in the attention mechanism. I thought it was very important that you understand it. So this lecture and the next lecture will be about position embedding. And today's lecture specifically will discuss about some common position embedding mechanisms. So in the last lecture, if we do a quick recap, what we discussed was multi-head attention. So we saw that you can calculate attention not just in a single head but you can have multiple attention heads operating parallely and finally the output from each of these attention heads are concatenated which passes through a mixing operation and then you have the final output but we also saw that this can be done through four loops you You could operate on
1:25head one, then head two, etc. Or you could do it using one single matrix multiplication. Both of this we implemented in Google Colab notebook also last time. Now today what we are going to do is, if you come back to this figure, which we will be referring to again and again in this lecture series, we will be for the first time focusing on the bucket number six. And bucket number six is about position. So, so far we have looked at attention score calculation, how tokens attend to each other especially when we discussed about mass self-attention, how attention heads can be multiple which was the multi-head attention lecture the last lecture and now the current lecture lecture number nine is about position information how do we inject it into the attention mechanism
2:12so today's lecture is about four attention mechanisms first is learned absolute position which is probably one of the most easiest way you could think of how to inject position information into the attention uh into the whole transformer architecture second is sinusoidal embedding which was famously used in the attention is all you need paper from 2017. this learned absolute embedding was used in gpt2 if i remember correctly and also in the bird paper and the third one is a relative third and fourth are actually relative position embedding so relative position embeddings means you look at the distance between the query and key rather than looking at where they are positioned absolutely in the sequence.
2:51So, that is also a very important way in which position embedding or position information is injected into the whole transformer architecture. So, we will discuss two architectures, relative position embedding proposed by Shaw and others and also something called as relative bias proposed in in the T5 paper which came out in 2019. These are all highly cited papers by the way. So, we are dividing the lecture into three parts. In the first part, we will discuss what is the problem if we do not have position embedding,ding what happens what is there an issue with respect to that in the second part we will see how absolute position information can be injected into the embeddings and third is if we
3:37don't want to do absolute what could be the reason and how will we implement relative position embedding so that's the last part of the lecture i will also walk you through a simple google collab notebook which shows you in presence and absence of relative uh in presence and absence of position embedding what the output of a small language model will look like because um obviously you should know that if position information is not available it's very difficult for tokens to attend to each other. You don't know, and this you will see when you are looking at two sentences consisting of the exact same tokens. I will show you an example in the upcoming slide. So I will show you using a Google Colab notebook
4:18with and without position embedding, how does the output quality look like? But before that, let's look at the problem. The problem is this, if I have two let's look at the problem the problem is this if i have two sentences this is the first sentence and this is the second one all now all dogs are animals versus all animals are dogs the first one is a true sentence second is not a true sentence but more than whether it is true or false what is important here is to notice that both sentences are consisting of the same tokens assuming one word equal to one token. Obviously tokenizer is different, but here assuming we are using a simple tokenizer where one word is equal to one token, this is two exact same sentences
4:55if you just look at it token by token. And for that reason, if you convert the tokens into corresponding vector embeddings, the vector embedding corresponding to the word all will be the same in both sentences. The vector embedding corresponding to dogs will be the same in both sentences. Although in the first sentence, dog is in position number two, if we assume position accounting is starting with from one, if we start the count from zero, it will be zero, one, two, three. But the position of the token dogs is different in both of these sentences but it wouldn't matter if position information is not injected the initial embeddings are exactly the same now what is the problem with this if embeddings are exactly
5:38the same what will happen is the dot products will also be exactly the same because we will take a token we will project it into query space and key space right and when we take dot product it is query dot key transpose right so if you consider dogs as the query in sentence number 1 and let us say animals as the key, it is basically Q2 dot product with K4 transpose. If you consider the dot product between query of dog and key corresponding to animals. If you do the same thing for the second sentence, it is actually q4 dot product with k2 transpose why because here dog if dogs is the query and animals is the key dogs is in position number four animal is in position number two now we know that the attention score has to be different in both of
6:38these cases because in the first sentence of the token animals is coming two tokens after dogs. So dogs plus one token plus one more token. That's when animals token is here. Whereas in the second sentence, the animals token is coming before the dogs token in the sequence. So the attention score has to be different because clearly the order in which tokens are arranged in a language matters. It changes the meaning, right? But if you take the initial embeddings like these if you take the embedding of dog and embedding of animals these embeddings are same whether you are in sentence number one or two which means when you project the embedding into query space and key space they will look exactly the same uh whether you are in position number two or position number four and for that reason q2 dot k4 transpose and q k4 dot k2 transpose sorry q4 dot k2 transpose
7:31will be the same number so the dot product between query and key will be the same as long as the query and key are originating from the same tokens their positions won't matter but this is not the right thing to do right you you know it logically you know that the attention score has to be different so what is missing here what is missing is the fact that these tokens are arranged in different positions so if we count the position starting from zero in the first sentence with the dogs with this token corresponding to dogs or with this embedding corresponding to dogs we have to add an additional position information in the form of a vector so we can do this in the form of input embedding plus a position information and there are also many other places where we could inject
8:18position information i will show you that in the upcoming slides but the basic idea is somehow we have to modify this input embedding so that position information is also part of it. Now if you are directly adding a position information into the input embedding, obviously this position vector should live in the same space as that of the original vector. So this will be a three-dimensional vector, this will also be a three-dimensional vector resulting in a new three-dimensional vector. Same thing applies for sentence number two. Dogs, the input embedding of dogs is added with position vector. And here this time it's position number three, this one. And that results in a new vector. So if you look at this vector versus the vector at the bottom, both are
9:01three-dimensional vectors, but and both are representing the input token dogs but the first one is dogs in the first sentence second one is dogs in the second sentence and they look different. Now a natural question that you should ask yourself is that you know when we are adding position information we are adding a vector right I mean in certain implementations we are adding a vector not in all implementations we are adding a vector right i mean in certain implementations we are adding a vector not in all implementations we are adding a vector to the input embedding but in several cases where we are adding input and we are adding position information we add it by adding a vector to the input embedding
9:34so how does it look like you have a vector corresponding to dogs and here is like a position embedding vector and this changes the resulting vector right now a natural question that you may have is when you add something and when the resulting vector is changing what if the resultant vector is landing in a semantic space corresponding to another word what if the original token which was corresponding to dogs when you add position information what is the guarantee that the resulting vector is not landing somewhere close to cats, which will totally change its meaning, right? So that's a valid question. Does this happen or does this not happen? So this is where we have to understand a few things about
10:16position embeddings. Firstly, I want you to become very comfortable visualizing the embedding space, not exactly how the embeddings themselves look like, but how are they spread in the space? Think about this. If you look at the number of tokens which were part of GPT's vocabulary, they range anywhere from around 52,000 all the way till there are bigger models with bigger number of tokens. So 52K, 52,000 tokens. So let's say we have total 52,000 tokens living in an embedding space of dimension 768. Do you think in a 768 dimensional space, 52,000 tokens can basically crowd the space? It actually cannot crowd the space. Here is a thought experiment. And this is also something which Jan Lacoon mentions regarding why, you know, in his JEPA model, which is unrelated to this,
11:12but for those of you who are aware of it, in JEPA models, one of the reasons why he says contrastive learning cannot work with negative examples is because negative examples cannot span this entire space. But anyways, here, this is what I'm trying to tell you. If you assume that your embedding space is two-dimensional and let's say you divide your x-axis into 10 parts, you also divide your y-axis into 10 parts, okay. Now let's say you want to span embeddings throughout this two-dimensional space you want to put one here so assume that you have a 10 by 10 grid so that every corner of your space is filled with some sort of embedding how many embeddings would you require you will require 10 raised to 2 embeddings because you will require a 10 embeddings like a 10 by 10
12:06grid right now this is because we have divided the space into 10 parts you could have divided it into three parts also no problem in that case you would have required three raised to two embeddings to fully span and crowd the entire space but now if you are looking at a 760 H dimensional space, if you are dividing every axis into 10 parts, you would require and by the way this is not from minus infinity into infinity, this is from whatever is the range of magnitude, there is a range of magnitude, it is not like embeddings can be like 10 raise to 13 or something, their magnitude is within a certain range. So, if you split that range into 10 parts within that range how many embeddings would you require that's the question here
12:49if you have 768 dimensional space you would require 10 raised to 768 number of examples that many embeddings you would require to really crowd your space this is if you are dividing the space into 10 parts if you were dividing your space such that every axis is divided into three parts you will require three raise to 768 that many embeddings now compare this 10 raise to 768 with 52 K the actual number of embeddings the actual number of tokens you have is only 52 K in let's say GPT small models. What is 52k? 52k is between 10 raised to 4 which is 10,000 and 10 raised to 5. Let's say it is 10 raised to 5 maximum. Compare the numbers 10 raised to 5 versus 7 10 raised to 768. If you had 10 raised to 768 you would really crowd the
13:41768 dimensional space but you still have a 768 dimensional space, but the number of examples here that you have a number of embeddings you have is simply only 10 raised to five. So no way your multi-dimensional space is crowded with large number of examples. In fact, it is a very sparsely distributed, all these embeddings are highly sparsely distributed. So because of that reason if you take an embedding of a word let's say this is the embedding of cat and if you add position information and let's say this becomes a new embedding for the cat with position information embedded it is very unlikely that this new position information injected version of the cat, it will land into the space for dog or it will land into the space for some other animal
14:29or some other object or something. This is one of the fundamental reasons. The space, the embedding space is very, very sparse. There is one more way in which you can visualize this. So, the reason why I am saying this is to tell you that the position embedding vector which we are adding in some of the position embedding vector which we are adding in some of the position embedding models, that vector is not a small vector. The magnitude of that vector can be maybe as big as the magnitude of embedding vector itself. So, it is not like you are making small change. The change that you are bringing to the original vector is high, but the resulting vector is pointing into a dimension of, into a certain direction of your embedding space where no other vector is pointing.
15:08And it is by the way seen that even after this change the closest vector to your position embedded vector is your original vector. The original vector if you look at the distance between the tip of the original vector and tip of the resulting vector after you add position embedding the distance between those two vectors will still be smaller than distance between the new vector and that vector for dog and the new vector and the vector for apple. So this new position embedded vector is still closest to the vector cat in 99.99% of the cases. There is one more way in which you can visualize the same idea. Assume that your embedding space is three-dimensional. What we are saying is, since the number of examples is very small, the total regions of the space our examples are spanning is very less. So, we can imagine that your third axis is not
15:58even being utilized. So, you have one example of a cat, which is pointing in this region of the space in two dimensions. Another example of, let us say, embedding of a a cat which is pointing in this region of the space in two dimensions. Another example of let us say embedding of a dog which is pointing into this direction. And you can imagine that when you add a position vector to cat, that position vector will make the resulting cat pointing also into the third dimension. So there is no way and there are no other vectors which are generally pointing in this direction which means the new vector is not really near the near any other vector embeddings. So, if this original one was that of a cat and if this was that of a dog embedding of
16:38a dog, the new vector corresponding to cat after adding position information it is still close to cat it is not close to dog even now. And this principle which you can visualize in small number of dimensions like two or three dimensions extends to larger number of dimensions also. In fact, associated with GPT paper, they had performed certain experiments and what they did was they took some 2000 example words and what they see was the closest vector even after and they had 128 positions, basically that many positions for position embeddings. And even after adding any of the position embedding to any of these words, the resulting vector in 99.999% of the cases is closest to its original embedding than to any other
17:25embedding in that space. One more thing that they found is the position vectors which we are adding it is only spanning primarily it is only pointing into 10 out of 768 dimensions available. And if you think about what about these 10 dimensions are the original vectors pointing into this these same 10 dimensions where your position vectors are also in general pointing to only 0.4 percentage of the spread of the original input embeddings are there towards the direction towards these 10 directions of the directions which your position vectors are pointing towards. So, it is very very unlikely that when you add position information to a dog, the dog will suddenly become cat that is very very unlikely that is that is the take away from
18:13this ok. Now, let us come back to the question how do we inject position information, where all can we inject, are we always adding a position vector to an input embedding or can we do something else. Look at this diagram you must be very very familiar with this diagram by this point in this lecture series. You have a vector, a word vector which you call as x, this is the input embedding. X is converted into query Q key K and value V by multiplying with query weights matrix, key weights matrix and value weights matrix so ignore this plus this r this r this plus ignore it you will see what it is in a moment then query and key you take dot product that is these are the attention scores which you convert by normalizing
18:55and by applying a mask if you want to have mass multihead attention you apply that and you take softmax to convert it into attention weights once you have attention weights you apply that and you take softmax to convert it into attention weights. Once you have attention weights, you multiply it with values. So attention weighted values are essentially the context vectors. So this output is the context vector. If you have n number of words at the input, n number of input tokens, you will have n number of context vectors. So this is the simple attention mechanism formula demonstrated in a visual way. Now this is one place where we can embed position information into the incoming words. You can
19:32simply add a vector corresponds to position. So this is what I mentioned earlier. You add a position vector to incoming embedding. This is one thing we will discuss today and this is part of you can there are two ways in which we can do that. We can add sinusoidal position embedding or we can add learned position embedding. The second place where we can do, which is part of Shaw embedding, which is a relative position embedding is we can do this with the key vectors. So we will have keys and we can add something to the key to say that with respect to query the key is this much far away it's plus two ahead or it is minus two behind something like that we can also add key embedding to value which i will show you in a moment so this also we will discuss sorry we can also add
20:19position to value which we will discuss another position embedding which is very very famous is rope or rotary position embedding which is applied to query and key. So, you rotate query and key before you take the dot product. We are not going to discuss rotary position embedding in today's lecture that we will discuss in the next lecture. So, that is one thing and alibi that also we will discuss in the next lecture. We will see how exactly Alibi modifies the query dot key dot product. And as I mentioned, this relative position embedding proposed by Shaw and others that also that is not just modifying the key, it can also modify value vectors. So, these are the position embeddings that we are discussing.
21:00One more position embedding that we will be discussing today is called t5 so t5 is actually not a name of a position embedding it's actually the name of the architecture which was proposed in the paper where they tried this so this is also a relative position embedding very similar to the the mechanism proposed by shaw but here you are not changing the key or value vectors but rather you are adding a constant uh you're adding a scalar to the query dot key dot product so obviously when you are adding something to the query dot key query dot key is already a dot product so when you are adding something to it whatever is being added is a scalar it's not a vector on the other hand when you are adding something to the key or when
21:39you're adding something to the value or when you are adding something to your incoming embedding all of those are vectors because when you add something to a vector, the added item also has to be a vector. On the other hand, rotary position embedding, it is a rotation transformation. So, we will see that in the next lecture. So, these are the position embedding mechanisms that we will be discussing in today's lecture and next lecture combined. So, first let us look at the absolute position embedding mechanisms, which is the part 2 of today's lecture. So, within absolute position embeddings, we will be looking at two mechanisms and the first mechanism is learned absolute positions. This was used in the GPT-2 paper. This was also used in the BERT paper, but by the way,
22:21it is also a very famous paper. Although it was not a generative model it was a masked language modeling based paper it's equally famous and in both of these cases the position information will be injected right at the level of input embedding so input embeddings you modify by injecting the position information all right so the idea of absolute position embedding using a learnable table is this you have have around 50,000 tokens, not 50,000. In the earlier example, I mentioned roughly 52,000 tokens, more or less like 50K, not like 52K. But anyway, it doesn't matter. 50,000 tokens roughly. And you have each token embedded in it as a 768 dimensional vector. So usually you convert your tokens into corresponding embeddings using this
23:08embedding lookup table. So you have a table which has 50,256 rows and 768 columns. Like that, you identify what should be the position embedding vector using a learnable table. So this table is learned through the process of training the GPT-2 model. It has 1028, four rows and 768 columns. 768 column, you might be clear why is it? Why is that the case? Because the embedding dimensionality is 768. So the position embedding also needs to live in the same space. So it has to be of 768 dimensions. But why is this 1024? If you have 50,257 tokens, don't you need that many rows? Don't you need 50,256, 57 rows in the position embedding table also? You don't need that. The reason is because of the context window size.
23:59Because in the forward pass of GPT-2, it is not going to process all the 50,257 tokens at once. Maximum it will process only 1024 tokens. So if the context window size is 1024 tokens, you only need to have information about that many positions because at any given point of time, you will have positions starting from 0 to 1023. So your position embedding table is also, it looks like a table like 1024 rows and 768 columns and what is done is if your if your token is if your word is let's say dogs Let's say this is the token you are looking at You will see where is the row corresponding to dogs and you will see that first you will convert this into a token id and then if the id is let's say uh 3800 you will
24:47look at the row number 3800 and find the vector corresponding to dog now if this token is appearing in the position number three in your uh input sequence you will look at this row you will look at this row of the position embedding table then what you do is these two vectors you will add with each other so the row corresponding to docs which is let us say the row number 3800 from your token embedding table and the row corresponding to position number 3 which is this row in the position embedding table there that you will add element by element so each dimension you will add up to get a resulting position embedding injected embedding which is of again 768 dimensions so this is how gpt2 works
25:34but you can obviously see the problems with this very soon this is just a visual showing the same thing this is the original embedding of the dogs showing in a lower dimensional space this is the learned position embedding if you are in position number one and this is the result versus if you are in position number three this is what you will add to the input embedding so when you add these two you get this when you add these two you get this okay now what is the problem with this approach the problem with this approach is that what if your context context window size is you problem with this approach the problem with this approach is that what if your context context window size is you are planning to increase the context window size
26:10what if you want to increase it to 1100 now you need additional position embeddings corresponding to 1024th position 1025th position up to 1099th position these additional positions which resulted from increasing the size of your context window you don't have any vectors any learned vectors corresponding to this in your learned position embedding table and this is this was one of the limitations of gpt2 and this is the limitation that sinusoidal position embeddings was trying to solve so in sine and cosine embeddings was trying to solve. So, in sine and cosine embeddings, you have a formula which may look intimidating when you look at it for the first time, but it is actually trying to do something very interesting. The interesting thing is, if your context window is increasing
26:57to beyond let us say 1000 or beyond 2000 or beyond 3000 etc., you should have the ability to produce unique position embeddings for those additional new positions right so if I increase my context window by another 500 I should be able to have a unique position vector defined for that particular position so the idea here is that if we can use a deterministic formula which will produce a set of numbers for each dimension of the embedding we can use that formula irrespective of whether my context window size is small or big. So I'll just show you with one example okay. So this formula you should also be able to see in attention is all you need paper where they mention what kind of position embedding are they using.
27:51So the formula is if we have a, let's say eight dimensional embedding for simplicity, you can calculate what the number should be for a given position for all the every single one of those eight dimensions. So, the goal is to produce a number like this. So, see this is a vector, an eight dimensional vector and this eight dimensional vector is corresponding to one particular position. What position is that? So, we are calculating this 8 dimensional vector for an example of position number 2. Position number 2 meaning token, the first token followed by the next token. So, position start with 0, so position number 0, position number 1, position number 2. And how do you come up with a vector that looks like this what's what what is the calculation
28:48behind this and how does what ensures whether if i change the value of the position whether the vector becomes another unique vector so this is the formula for for your vector the value depends on whether the position is an even number or odd number. What are even numbers? Not the position, whether dimension is an even number or odd number. So if I have an 8 dimensional vector, the dimension 0, dimension 2, 4, 6, they are odd dimensions and sorry, even dimensions and one three five seven these are odd dimensions so if it's an even dimension the formula that is used to calculate the corresponding value corresponding position embedding value for that dimension is sin of position divided by 10000 raised to 2i by d. A fairly complex, I mean complex looking formula you do not
29:51know why there is 10000, what is this 2i by d lot of questions you may have. I will show you one graph immediately after I show you this calculation then the graph will make a lot of sense. So, first let us plug in numbers, what is the value of position? Cos position equal to 2. So, the value of position is equal to 2. 10,000 let it be there, 2 i by d, what is 2 i by d?. By the way please note that this formula applies only to dimensions that are of even values. If dimensions are of odd values the formula is basically cosine of position divided by 10,000 raised to 2i by d. Now let us look at by starting with i equal to 0. So if i is equal to 0, the formula for the even position is this. So 0 is an even, not position, 0 is a dimension that is even so 2i by d value will be 2 times 0 divided by d
31:08where d is equal to 8 that will be 0 but when you jump to here you should not you are not increasing the value of i instead you are now looking at the other formula cosine formula so basically if you look at this whole table these are the eight dimensions and these are the values of the position embedding corresponding to those eight dimensions for position equal to two. It is an example. You can calculate this for position equal to zero, position one, position three, any position. Now, these are the ones shown in circle are calculated using the sine formula. The ones shown in squares, which are odd dimensions are calculated using the sine formula, the ones shown in squares which are odd dimensions are calculated using this cosine formula.
31:50So first let us focus on the sine formula, when i is equal to 0 you get 2i by d equal to 0, when i is equal to 1 you get 2i by d equal to 2 by 8 which is 1 by 4 that is 0.25, when i is equal to 2 you get 4 by 8 which is 0.5 and when i is equal to 2, you get 4 by 8, which is 0.5. And when i equal to 3, you get 6 by 8, which is 3 by 4, or it is also 0.75. Same thing also applies for cosine, because in cosine, the value of 2i by d is the same. It is just that for even position, for an odd position, the formula has a cosine value for even position it's a sine value so once you have 2i by d you can calculate the the divisor so divisor is nothing but 10 000 raised to 2i by d what is 10 000 raised to 2i by d first of all 10 000 is 10 raised to 4 and 10 000 raised to 2i by d
32:40is 10 000 raised to 4 the whole raise to 2 i by d and we can rewrite this as 10000 raise to sorry 10 raise to 8 i by d right and this part is very easy now so when i equal to 0 and if damn and d is 8 right so basically this is 10 000 raised to 10 raised to um i so when i equal to 0 the divisor is 10 raise to 0 which is 1 when i is equal to 1 the the divisor is 10 raised to 1 which is 10 when i equal to 2 it will be 100 when i equal to 3 it will be 1000 so if you look at this sine and cosine formula you can you can see that these two dimensions so this is a pair of odd and even and odd dimension they have the same divisor these two dimensions they have the same divisor. These two dimensions, they have the
33:45same divisor, but the value of divisor has increased. These two have same divisor, these two have same divisor, 10,000 raised to 2i by d. And now, what is this total value inside cosine? It is position divided by 10,000 raised to 2i by d. So, the value of what is there in the sine and cosine formula is given in this table. So, the value of the dimension number 0 for position number 2 is sine of 2, sine of 2 is 0.91. For the second, for the dimension number 1, you have to switch to cosine formula. It is cosine of 2 which is minus 0.42. For third one, it is sine of 0.2, cosine of 0.2 and you can imagine why sine of 0.2 is 0.2 itself because sine of a small number approximately equal to that that number itself so here it is
34:33sine of 0.02 it is that number itself now let me show you instead of showing you a big table i'll first show you a graph then i'll come back to this okay now this graph explains what the position embedding for given for any position for any dimension looks like so here I am only plotting sine graphs the cosine ones I am not plotting that's the reason why here you can only see dimension 0 dimension 2 4 and 6 but dimension 1 dimension 3 dimension 5 dimension 7 also is needed but dimension 1 is basically if this is the sine graph with the same frequency you will have a cosine graph first of all just think about it for dimension 0 what is the what is this graph what is the x-axis x-axis is position you need to create position embedding
35:26for let us say if your context window size is 1000 you need to create embeddings for all 1000 positions and in this case we are assuming that we are having an eight dimensional vector okay so what will be the vector embedding corresponding to position number 0 the vector embedding will be collection of all these values the strip of the first you know you see these black dots collection of those values of course for odd dimensions you will have the these dots picked up from the cosine graph these are only showing the even dimension you also have the odd dimensions obviously those two dimensions you need to construct the full eight dimensional uh vector embedding for position zero for position zero meaning for whichever token
36:10comes at position zero this particular embedding vector will be a position embedding vector will be added to it look at the next one or if you consider position let's say one two three four five six for position six your embedding vector would look like a collection of these dots The numbers resulting from these dots And these are for the even dimensions you should also consider the numbers coming from the cosine graphs of these corresponding graphs Which are the odd dimensions? Now what is happening is when you are you know? Although this is repeating frequently, the numbers in this graph is repeating more frequently. So you might think that, oh, for my dimension zero, the numbers might repeat because this
36:56graph has a very high frequency. But it does not mean that your embedding will repeat. Your embedding has to be unique for no two positions should have the same embedding will repeat your embedding has to be unique for no two positions should have the same embedding and you know what ensures that you look at the later graph so this is actually not a straight line this is actually a sinusoidal graph with frequency very very small the dimension number four it is actually a sinusoidal graph with an even you know with a slightly higher frequency than this previous one dimension two is also a sinusoidal graph but you know a small frequency so what is happening is even though the values in this dimension 0 may repeat for position 0 and let's say position 1000 that won't happen
37:38for later dimensions for later dimensions it takes a long time before the value repeats because the the frequency is very small which means the vector itself won't repeat because of uh having such a low frequency in in higher dimensions so this graph is like a very clever mathematical way to construct embeddings that are unique are they unique in every single dimension no not necessarily but overall as an embedding it will be unique because there will be some of the dimensions which are completely unique. So this dimension 6, for example, here, the values will never repeat. But in dimension 0, some value will repeat. So here it is written. It repeats every 6.3 positions. So by the time you track around 20 positions, when you go from position 0 to position 20,
38:26three times the same value would have repeated. So, if this is the value, this value would have repeated here, that value would have repeated here, but that value would have repeated here, rather four times. But if you look at the same thing in dimension 2, it repeats only once in every 62 positions. So, for a number in dimension 2 to repeat once you start with position 0 you have to go to the 60 second position only then a number will repeat so yes in some dimensions the numbers repeat but overall when you look at the overall position embedding vector even though the dimension 0 may repeat in some of those vectors, the dimension 2 has a lower likelihood to repeat, dimension 4 has an even lower likelihood to repeat, dimension 6 has very low likelihood
39:11to repeat and if this is a very high dimensional space, the chances of the dimension repeating itself becomes very very low. And if I go back to a graph which I had shown in the first slide of this lecture, I do not know how many of you noticed it if you look at this graph this graph shows exactly that just just look at this graph this color map is literally showing the position embeddings of a 32 dimensional vector for 48 positions so this is position 0 um this first first row position 0, let me open the file. The second row is position 1, position 2 and up to position 48 and each row is a vector, 48 dimensional vector. And look at the first dimension, in the first dimension you can see values are changing
40:01rapidly as you see from this color map. Whereas in later dimensions the rate at which values are changing rapidly as you see from this color map whereas in later dimensions the the rate at which values are changing is very low it may look like the colors are the same actually there is a fade change in the color which you can't perceive like for example if you look at this one the color is slowly changing just like how the wavelength of the graph is very big the same color has not repeated in throughout this column anywhere because the color is still gradually changing. But maybe in the first column, the color has repeated multiple times. In the second column, maybe color has repeated maybe slightly lesser
40:34number of times. In third column, also the color may have repeated once or twice. But as you move to the rest and rest of columns, no color will repeat. The color is slowly changing. And here you might think oh this is entirely white but that is because we have limitation of showing the color but actually what's happening is that the color is faintly changing and as a result if you look at any given vector so this is the vector for position 0 and this is let's say the vector for another position. You may see that for this dimension and for this dimension the values might be the same. For this dimension and for this dimension the value might be the same. But for the rest of the dimensions, although they look similar, they are not exactly the
41:20same. The values are, there is subtle difference in the values. So, this is how unique position embedding vectors you can create for large number of positions. So, this can handle even, you know, very big context windows. So, let me now go back to our original slide. We were at slide number 20. Yeah. Okay, now let us go back to this previous table. Now you should be able to have, you know, visualize how position embedding is calculated. So now this is for position 0, position 1, etc. up to position 5. I could have shown up to position 8, but then the page will become too crowded. So here we are showing basically a 6 dimensional vector. And each row is one vector. The first row is the position, is the vector corresponding to position 0.
42:13Last one is vector corresponding to position 5. And you can see this, the dimension 0, dimension 1. So, dimension 0 is sine, cosine 0 is sin cosine sin cosine sin cosine sin cosine and if the sin value is 0 corresponding cosine will be 1 if sin is 0 cosine will be 1 sin is 0 cosine will be 1 so the square root of this the square of this plus square of this will always be 1 because sin square plus cos square of one number is always equal to 1. So, you will see 0.96 square plus minus 0.96 square plus 0.28 square, we can do it 0.96 raise to 2 plus 0.28 raise to 2, this should be approximately equal to 1, it is equal to 1, right. So everything checks out mathematically and here you see that and by the way these numbers
43:14are rounded off numbers, okay. Like so here in for example in dimension 5 you have 0.04 here and it is this is 1, but actually this is not 1, this is 0.99 something but i have i have just written it in the rounded off manner but of course this number is unique this number is not exactly equal to one so when you see it it may look like exactly same number but it's actually not an exact same number and how is this calculated this is this number what formula did we use to calculate sign of position what is position 0 divided by 10 000 raised to so 10 000 is what 10 what 10 raised to 4 raised to 2 i by d what is 2 i 2 times 0 and what is d d is the dimensionality which is 6 in this example. So, 2 times 0 divided by 6, that is how this was calculated. How is
44:06this calculated or no not that let me pick another one. How is this calculated? That is a which dimension? It is dimension number 3 which is odd number. So, it should be a cosine formula that is cosine of position, what is position? Position 4. So, 4 divided by 10000 raised to 2i by d, sorry, yeah, 2i by d. What is the value of i? Value of i, because this is position 2 and 3 is equal to 1. So, 2i will be 2 times 1, which is 2 itself and d is 6, raised to 2 by 6. So, this is the formula which is selected in this. So, you can calculate this is how you can calculate the position embedding vector for any position and what do you do once you have the position embedding vector same thing which you
44:56did earlier you have your original embedding to your original embedding you will add your position embedding vector and this will give you the final embedding all right so that is the sine and cosine embeddings and this was used in gpt paper and i'm sure we can come up with probably other formulas also but look at what is happening when we add this position embedding you have word, the original input embedding plus corresponding position embedding. And you multiply this with query weights matrix and this is the query. This resulting thing is the query queue. Similarly, you have input embedding plus position information added. And that, this is key, right? That is multiplied with a key weights matrix to get the key so this
45:48is basically q i and this is k j key corresponding to the token number j but when you multiply this if you expand it you will see that earlier if position embedding was not there you will simply have these two terms you will have the word transformed into query using query weights matrix the word j transformed into key using key weights matrix but now here we will have additional three more terms where the position term is coming into picture so here position is coming into picture here position is coming into picture, here position is coming into picture and here also position is coming into picture. So now in the query.key transpose, which is the attention score formula, attention score calculation, originally if position embedding was not there, it will simply be the transformed
46:38versions of word, basically query and key query.product with key, that's all. But here it won't be just the query and key resulting from the original token it will be original token plus position converted into query and key when you take the dot product you will basically you can imagine there are these three additional terms and because these three additional terms are there obviously your query dot product with key which is the attention score will be different so this is how the earlier problem which we mentioned is solved earlier we mentioned that all dogs are animals and all animals are dogs. When dogs is the query in both of the sentences, the attention score and let us say animals is the key, the attention score has to be different when you take dot product between query
47:20and key in those two sentences and this is indeed how it happens. Now let us move on to the part 3 of the lecture which is relative position embedding and this part is fairly easy. So the idea of relative position embedding is the following. You may have a very big context window and in the context window maybe you have two tokens that are appearing such that they appear close to each other or they appear in two relative positions that are the same. For example, in the context window, the dog chased this kind of term can appear at an earlier position, position 1 and 2 respectively. And the same thing can appear sometime later in the context window also. In the same context window, let us say in position 101 and 102 102 also let us say the same thing is appearing.
48:07So what really matters is sometimes to capture the meaning of words what you are not looking at is whether this word is in the 100 position or not. What you are really looking at is what are the other words that are close to this word. So the relative position between the words plays a big role in how humans understand the meaning of a sentence. It's not just about if you have a very long sentence or if your context window is very big, you can fit multiple sentences in the context window. But if you really look at how humans are understanding the meaning of sentence or meaning of groups of words, the way we understand it is we see the words and we look at what are the other words next to it.
48:49So the relative position between the words within a sentence or within a context window might have more importance compared to absolutely where are they present. Are they present in the second half of the context window or are they present in the first half of the context window? Maybe it does not matter. This probably does not matter in deciding the meaning or the context of the words, right. So, that was the idea behind people thinking, maybe we should define relative position embeddings. So, the first relative position embedding was introduced in this paper by Shaw and others, self-attention with relative position representations. Ashish Vaswani, the name you will recognize, he is part of the attention is all you need
49:28paper. So, where is this position embedding injected? Where is it being injected? It is being injected primarily into the key of the key corresponding to the token. So, you should have many questions like why is it only into the key, why not into the query, why not into the value, how does it etc so let's look at that so relative position embedding uh in this shaw at all paper it only restricts to primarily five position embeddings five learned position embeddings okay so if let's say this is the query, if we are considering query corresponding to dream and let's say these are all the keys, basically token converted into query and token converted into key, the distance between dream as a query and dream as a key is zero.
50:17So we need something, we need a learned vector for distance equal to zero. So we will have a learned vector. That vector will live in the same dimensional space as that of query key value space. So if query key value space is of 384 dimensions, you will have a learned vector corresponding to relative position of zero, which will live in the 384 dimensional space, right? So as you can see here for all the diagonal elements, the relative distance between query and key is zero. So all of these will have the same learned vector. What about the distance between dream as a query and big as a key? So big is coming right next to dream. The position of big is plus one or to the right side of dream. So this is classified into a bucket which we can call as a plus 1 bucket, but that is
51:13actually literally a new learned vector corresponding to distance equal to plus 1. Now here I have written it as plus 1 in blue color, but it does not mean that it is a number. What I am trying to show is there will be a vector living in 384 dimensional space or whatever is the dimensionality of the query key value space. In that vector, there will be a learned vector which will be learned during the process of training this neural network and that vector will apply to those query key value pairs whose distance is plus 1. So if I ask you what is the distance between AND as a query and WORK, the distance is plus 1 as written here. And which will be the vector corresponding to that that we will pick? We will pick this vector. So now all of these query key
52:01value, sorry query key pairs belong to relative distance of plus 1 and they will all map to you know the position embedding corresponding to all of those query key pairs will be position corresponding to this plus 1 the learned vector corresponding to plus 1. Same thing applies for minus 1 what is minus minus 1? Minus 1 is if the key, if the position of the key is to the left of, is one step to the left of the query. So if dream is your query and or rather if big is your query and dream is your key, dream is coming before big, which means to go from query to key, you are taking one step backwards. So we denote all of those query key pairs as minus one, and for corresponding to minus one also,
52:54you will have a learned vector. Now this much is okay, right? Now is the interesting part. Look at the plus two, for everything else, you are assigning plus 2 distance. So what we are doing here is whether the distance between query and key is plus 3 or plus 4 or plus 5 or plus 2 in all those cases the distance we are capping at plus 2. Similarly if the distance between query and key is minus 2 or minus 3 or minus 4 or any other negative big number we are capping it at minus 2. What we are saying is, if the relative distance between two tokens are too big or too small, we do not take that huge relative distance, we cap it at a maximum number or minimum number. And there will be a corresponding learned embedding, learned position embedding corresponding
53:42to that. So, totally how many learned position embeddings will be there? One corresponding to 0, one corresponding to minus 1, one for plus 1, one for minus 2 and one for plus 2. So, total 5 learned embeddings are the only thing required, irrespective of whatever is your context window size. Because for a given query, if you look ahead or if you look behind, you are only considering maximum plus 2 distance ahead or if you look behind you are only considering maximum plus two distance ahead or minus two distance behind so for any given query you just need five relative position embedding vectors and because this is relative position embedding vector with respect to query you don't have to modify the query itself because you are you are saying query is my origin i am looking from the point
54:25where query is and i am just trying to see where my key is is the key plus one distance away is it plus two distance away or is it at plus zero distance away or is it minus one minus two where the key is so the only parameter that we have to modify here to modify the attention scores is key you don't have to modify the query so what is done is in the equation when you calculate attention score you are basically adding a learned vector to the key you are not adding it to the query so basically query dot product with key plus the learned vector what is the learned vector learned vector whatever is the learned vector we pick out of these five vectors it will depend on What is the learn vector? Learn vector, whatever is the learn vector, we pick out of these five vectors,
55:06it will depend on what is the distance between query and key. And we can also have a learn distance between query and value, which we can add to the value also. So in Shaw paper, we can do that also. That I had shown you as a graph in the very beginning. So for example, this is how you will use it how you will implement it so the sentence that we were looking was dream big and work for it and so if we are considering query as it okay so if I write the sentence here, green, they are a big and work for it. Now if query is it and key is and, and is here, what's the distance between query and key? It is minus one, minus two, minus three, but minus three we will not use. We will clip it at minus two, right?
56:04We say we are maximum considering only up until minus two and corresponding to minus 3 but minus 3 we will not use we will clip it at minus 2 right we say we are maximum considering only up until minus 2 and corresponding to minus 2 there will be a learned position embedding vector here in this case it happened to be something that looks like this so that learned position embedding vector corresponding to minus 2 position distance we will add with the with the key to get the new key so this is now the new key. And query is not modified. Query is still the same. And when you take the dot product between query and key, you will get a score of something like minus 0.59 because dot product is basically a scalar.
56:33What if you consider query as big and key as and? So big is this and is this. What's the distance between them? Distance between them is plus 1. So if the distance between them? Distance between them is plus one. So if the distance between them is plus one, you will have a distance. You will have a position embedding vector corresponding to plus one distance. That is this. That you will add to the key. You won't add it to query. So what is the key? Key was and. So this is the key. You are adding the position vector along with the key. So this plus this. Now this is the new key. And that key, you will take dot product with the query, unmodified query, this is the unmodified query to get the dot product which is the scalar dot product
57:14query dot key transpose which will be here, this number. So this is how the key is modified in relative position embedding introduced by Shaw and others. There is one more position embedding relative position embedding which I want to discuss in this lecture and that will be the last one today is relative bias introduced in the T5 paper. So it is the paper is called T5 because it's transfer learning with unified text to text transformer transfer learning or something. I have written it in one of the slides. Yeah. So text to text transfer transformer. So TTTTT. So T5. So that's the reason why this paper, I mean, the model in this paper or the architecture in this paper is called T5. But architecture is not what is mattering.
58:05What matters here for us is what kind of position embedding they used. So the position embedding was looking like this. So interestingly, this time the position embedding is added directly to the dot product between query and key. So you might wonder, oh, how is position embedding added to a dot product because dot product is a scalar right and so far we have been discussing about embedding in position embedding in terms of a vector. How can you add a scalar a vector to a scalar you cannot. So actually here what is added is not a embedding here what is added is a number. So for certain positions we calculate a number that number is added into the dot product between query and key. So now the question is, how are we considering relative position bias in this particular
58:53example? And that's the reason why it's called as a bias, right? Bias is usually one number. So that bias is helping us change the value of the dot product between query and key. So how is this bias calculated? Let's see. So the idea is very similar to what you saw earlier. Okay, so don't get intimidated by seeing a lot of these numbers. It's fairly straightforward. Look at the distance first of all, the distance between query and key. How many distances are possible? So if query is 3, the distances possible are you can have 0, plus 1, plus 2, plus 3, plus 4, plus 5 in this particular case. Similarly, if query is it, compared to it, you can have minus 1, minus 2, minus 3, minus 4, minus 5, 0 also is there.
59:40So here the range of values are basically for corresponding to one particular query, you can have distances in that range. So, if the key is belonging to the left side of query, so if the distance is a negative number, so for example, if distance is minus 1, minus 2, minus 3, minus 4, minus 5, etc. We will consider these buckets. Okay, so basically for all negative numbers, we will consider bucket number 0 to 15. For all positive numbers, we will consider these buckets, bucket number 17 to 31. What are these buckets? Previously, we had 5 buckets. Bucket for plus 2, minus 2, plus 1, minus 1 and 0. Similarly, here we are having 32 buckets. So instead of 5, we will have 32 buckets. What does 32 buckets simply mean? 32 buckets mean for each bucket, there will be one number
1:00:40that is learned as the relative position bias. However, one more additional thing is this is implemented per attention head. So for each attention head, you will have 32 buckets or 32 numbers that are learned, learned relative position bias. So if you have 10 attention heads, you will have 32 multiplied by 10, 320 unique relative position number, position bias numbers learn. But how are we applying this and how does this and to which kind of query dot key dot products we are applying this? This is how. So if you look at the, if you look at and as the query and big as the key, what's the distance between and and big? If and is the query and big is the key what's the distance between and and big if and is the query
1:01:27and big is the key the distance is minus one so if the if the distance is minus one uh it is basically the the absolute magnitude of distance is basically one right if you if you forget the sign and since the key is coming before the query since key is in position number one and query is in position number two you are now looking at these blue buckets from the blue bucket if the distance is one you will consider bucket number one and corresponding to bucket number one there will be a learned uh learned relative position bias a number a number like 0.63 something like that and for all such positions so now if you look at the if you look at all these numbers these are all cases where between query and key
1:02:13the distance is minus 1 but I am showing here 1 because minus 1 is mapping to bucket number 1 these are all pairs of queries and keys where the distance between query and key is minus 2. But if it is minus 2, it maps to bucket number 2. Similarly, if it is minus 3, it maps to bucket number 3. If it is minus 4, if the distance between query and key is minus 4, it maps to bucket number 4. Similarly, look at the case where you have distance equal to zero. The reason why we don't have bucket 16 is, if the distance is zero, you don't need two buckets, you just need one bucket, because there is no plus zero and minus zero. So you just need only one bucket. So for that reason, if the distance between query and key is zero, which means if query
1:03:04and the key are originating from the same tokens, the distance between query and key is zero which means if query and the key are originating from the same tokens the distance between them is zero right if it's the same token you only focus on bucket 0 bucket 16 does not exist now what if the distance is plus 3 what would be the bucket if the distance is plus 3 the bucket would be bucket number 19. so if the distance is plus 3 the bucket would be bucket number 19. if if the distance is plus 3, the bucket would be bucket number 19. If distance is plus 2, you will look at bucket number 18. So here 18, 19, etc. are not distances. They are also not the numbers that we are adding. That is just the name or just the ID of the bucket corresponding
1:03:42to that distance. But each of this bucket will be associated with one scalar so bucket number 31 might be associated with a scalar like 0.613 bucket number 27 might be associated with a scalar minus 0.132 and how and the way we decide whether this should be the relative position bias or this should be a relative position bias depends on whether the distance between my um my query and key is belonging to this particular bucket and by the way why are we calling this as a bucket this is the reason if the distance is one two three four five six seven or zero you have unique buckets for each of them okay however as you move farther away if the distance is eight nine 9, 10 or 11, they all belong to this bucket number 8 or bucket number 4, depending on whether it is plus 8 or minus
1:04:318. So, if it is minus 8, minus 9, minus 10, minus 11, you look at bucket 8. If it is plus 8, plus 9, plus 10 or plus 11, you look at bucket number 24. And you can see that these buckets are widening. This bucket is 46 to 63. This is 64 to 90. The last bucket is 91 plus anything else. So if your context window is 1000, let's say the distance between query and key is 300 plus 300. Your key is 300 tokens ahead of your query. What would be the bucket that it belongs to? It will belong to bucket number 15. Sorry, sorry, sorry. Not it will belong to bucket number 15. sorry sorry sorry not bucket number 15 bucket number 31 what if your uh key was 200 tokens behind your uh query since it is 200 tokens behind the query the number that you are looking at is minus 200 and since my here you
1:05:20are saying for 91 and above if the absolute distance is 91 and above and if the key is to the left side of the query you look at bucket number 15. so basically you are you are having a logarithmic division of your distances of your relative distances you are bucketing them and this this creates relative distance um relative distance bias scores which are trainable and that is also calculated per head if you have 100 heads if it's a 100 head multi head attention mechanism you will have 100 multiplied by 32 that many buckets or that many relative position bias numbers and once you have the relative position bias number where do you add it you simply add it to the dot product between query and key i'll show you exactly where so So if this is our sentence, dream, begin, work for it. We can consider, let's say it as the
1:06:11query and key can be, let's say any one of this. So if it is the query, query is over here, right? The distance between it and four is minus one. The distance between it and work is minus 2. Between that it and and is minus 3, minus 4, minus 5. So minus 1 belongs to which one? Bucket number 1. Minus 2 belongs to bucket 2. Minus 3 belongs to bucket 3. If it was plus 1, it was bucket 16, sorry bucket 17. If it was plus 2, it was bucket 18. So here the buckets that we will have is 5 4 3 2 1 and each of this bucket will have a corresponding learned relative position bias so these are the learned for relative position bias corresponding to these buckets that bias we will add to the dot product between query and key and what is the dot product here the dot product between query and key in this
1:07:01case is 0.9 what is this dot dot product of query corresponding to it and key corresponding to dream is 0.9. 0.9 plus minus 0.8 is equal to 0.1. So this is a new attention score between query and key. Which one is a query? Query is it. Key is dream. How does the dot product change when you are looking at AND as the key? If AND is the key, the relative position bias is minus 0.2. So 0.7 plus minus 0.2 equal to 0.5. So now these are the attention scores between query as it and all the keys before adding the relative position bias. after adding the relative position bias, this is how it would look like. So relative position bias introduced in the T5 paper actually does not introduce a position
1:07:56embedding vector, it rather introduces a bias score which is just a scalar. So that is all I wanted to discuss in today's lecture. So in summary, we have looked at learned absolute position embeddings there were two of them one was learned absolute and one was sinusoidal learned absolute introduced the introduced the embedding table but the the downside of that was if the context window size increases you won't be able to you know create position embeddings for that sinusoidal position embeddings have the ability to because it uses a sine and cosine formula you can handle very big context windows then we also discussed two relative position embeddings one was introduced in the paper by shaw and others the second one was introduced in the relative position t5 paper
1:08:41both were relative position embeddings sorry not embeddings third one was a relative position embedding fourth one is actually a relative position bias it's not an embedding it's just a number it's just a bias okay and where were these things added the learnt absolute embedding is added to input embedding x sinusoidal embedding is also added to input embedding x relative position embedding is added to key. It can also be added to value. It is not added to query. Relative bias is added to the dot product between query and key. It is not added to neither the key nor the queue because it is not a vector. It is a scalar. So this brings us to the end of this lecture. I hope you have a fairly decent understanding of position embeddings and how they are implemented
1:09:28in practice. One last thing I want to show, which I had promised at the beginning of the lecture is the output quality difference between the cases where you have position embedding and you don't have position embedding. So I will just show you that as the last thing. And what we want to cover in the next lecture is rotary position and embedding an alibi so we will discuss that in the next lecture but let's also spend a few minutes on the google collab notebook now so this is the google collab notebook i don't want to spend any time going through the code at all because that is not my intention here i just want to show you one demonstration that's all okay. Okay. So here I'm showing you two cases. It's
1:10:06a tiny GPT-like model trained on a dataset called as tiny stories. So tiny stories is a very popular dataset, very short, like paragraph level stories. So it's not even like a story. It's like few sentences, which we call as a story. Now, transformer architecture, let's not spend any time on it because that is immaterial right now. I mean, it's a very, very small transformer architecture. But what I want to show you is one thing, which is this graph. So this graph shows as the training proceeds, how does the test loss look like with and without position embedding. So you can see that the gray color graph is without position embedding and blue color graph is with position embedding. You can see that when you have position embedding, when you inject position information, your network trains better.
1:10:50Surprising thing here is that the loss is not terribly low in the absence of position embedding because network can learn a few things even if you don't inject position information. But network won't be good enough. Although this graph does not look very different, the output I just want to show once. I'll share the link to this Colab notebook with you, so you can spend a lot of time on this notebook. But here are two outputs. The first output is without position vector and second is with position vector. You just read the stories produced by these two models and tell me which one looks better. First one reads, once upon a time there was a little bird named Lily. So far good. She loved to put, she loved to put it, it was a little bird named lily so far good she loved to put she loved to put it it was
1:11:26a for t one day the sun they both say it was so play so it's not making any sense right there is no meaning whereas with position vectors it's slightly better once upon a time there was a little little bird named lily she loved to put it it on her bed not took a big castle one day she decided to go to the park and they put the fox so story is still not making sense there are grammatical mistakes and all but slightly you can see that the second version is slightly better uh on the other hand on the first one without position embedding one day the sun they both say it was so play with the with the should sometimes so it's like just a collection of words no meaning whereas in the second one some meaning is there as she could have some special to park special to the
1:12:10park to put it away i mean the result is so bad because the the number of training steps i have run is very little because i i here we are not training and you're uh we are not training an llm or slm right now here we are just showing the difference in the case where you have positioning information you don't have she was so happy to have a big fun playing in the park some sense is there so when the position information is injected the output becomes more sensible as you can clearly see here and um obviously like i mean i don't even have to show you this but this is just a visual proof of what we have been talking about so far the importance of position embedding so that's all I'll just share
1:12:47this link with you just take your time to go through it no rush and we will anyway if in the future lectures if our scope allows we can also go through some Google Column or collab notebooks in detail that goes through the attention mechanism itself so far we have not gone through any implementation detail of attention mechanism. We have looked at some of the self-attention implementation classes in some of the previous lectures, but that is the extent of our code so far in this lecture series. So, thank you so much. This is a good time to end the lecture. We will meet again in the upcoming lecture on rotary position embedding, alibi etc. And rope is by the way a very famous, very popular position embedding which was also used in deep seek model.
1:13:31So you will enjoy learning about it. Thank you so much. See you soon. Bye.