Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2

0:00Hi, welcome back to the attention family lecture series. In this series we will be discussing various different attention mechanisms starting from the very early scale dot product attention all the way till the latest attention mechanisms that are proposed in papers including chemi delta attention and various other attention mechanisms that are extremely popular. However, before jumping into different attention mechanisms or even the basic foundations of attention mechanisms, it's very important for us to understand what is the relevance or what is the significance of attention mechanism in the first place, what existed just before that, and what were the shortcomings of the architecture which was

0:40introduced before attention mechanism was in existence and how does attention mechanism help us make the process of let's say language to language translation or paraphrasing or even text generation etc better. So today's lecture is about what existed before attention and primarily we will be discussing about neural machine translation or sequence to sequence models which had a major bottleneck however i thought of including this lecture as one of the foundational lectures in this series because this lecture will teach you a few different concepts like embeddings context vector etc so i thought it is very important for you to have a mental picture of what the word context vector means because this word will be using in

1:25almost every single vector from here on almost every single lecture from here on so let's get into the lecture so a quick recap of what we discussed in the last lecture last lecture in the last lecture I introduce you to what this series is about I literally named around 37 different attention mechanisms and although I did not explain any of them because that was not the intention of the last lecture it was just to show you just the sheer volume or sheer number of various different attention mechanisms and there are many more there are 50 plus very popular attention mechanisms but in this series we are not going to be randomly studying these attention mechanisms as i have promised in the last lecture what we are

2:04going to do is we will look at attention mechanisms but we'll try to classify them or we'll try to bucket them and i had shown you one figure with seven different buckets which i will again show you in today's lecture at the start and our goal is to understand attention mechanism from the perspective of okay how are these two attention mechanisms different? Is it in the way the tokens are attending to each other? Is it in the way we deploy this on GPU? Is it in the way the fact that attention, one of them is scaling its order of magnitude of computation with square of number of tokens, the other one is less than n raised to 2. Is that the difference? So this picture we should have in our mind by the end of this lecture series,

2:46so that even hopefully one day you might be able to come up with your own attention mechanism that is better than what is existing today. So today's lecture is not about any of these buckets yet, because we are yet to get to the attention mechanism. Today's lecture is, why did people think of creating this thing called as the attention score so attention score is in the bucket number one so this is the picture that i had shown you in the last lecture we have seven buckets first bucket is about scoring how exactly is attention score calculated second is about connectivity which tokens are attending to allow to attend to which other token third is about attention heads and kv cache fourth is about uh you know having quadratic or sub quadratic uh order of magnitude of complexity

3:31when it when it comes to attention score calculation fifth is with respect to implementation of this in gpu this is where flash attention comes into picture sixth is about injection of position embedding or position information into the attention scores or into the context vectors. So that will, this is not the sixth lecture by the way, this is the sixth bucket, okay? So we may discuss about position embedding in one of the earlier lectures. It's not necessary that we have to discuss position embedding towards the last part of the series. That may not be the case. And we will also discuss about attention mechanisms from the perspective of whether it is applicable to text or graph or video or image whatever the domain is okay so before we get into

4:13the attention score calculation part which primarily comes in the next lecture we have to understand before this what existed and why did we think of something called as the attention score? So today's lecture is divided into three parts. In the first part, we will discuss the translation or neural machine translation, which was basically the architecture that we could use for converting, let's say, English language into French or vice versa. And how this architecture had a encoder decoder setup and what the basic architecture of NMT or neural machine translation was and between the encoder and decoder the information was passed in the form of a vector called as the context vector. So we will also discuss regarding this context vector and how this one single vector carries

5:00the entire information present in a sentence whether it is a long sentence or short sentence and. And of course, from that itself, you can imagine what the difficulty is, because if it's a long sentence, this one context vector cannot possibly hold that much information. So that is an issue. And we will see, we'll discuss about that. And we'll also discuss how this context vector is constructed. And in the last part of the lecture, we will discuss why this kind of a context vector based encoder decoder architecture, which was used in neural machine translation before attention mechanism and that was using a architecture called recurrent neural networks, RNNs. And we will discuss why this was breaking.

5:41Breaking meaning if the sentence length is longer than a certain threshold like more than 20 words or so you could see that the translation quality drastically drops down drastically decreases so these are the points that we will discuss in today's lecture so welcome to the lecture so first let us look at the architecture behind the neural machine translation or sequence to sequence model so here is an example where you have an english sentence this is a car converted into its french equivalent you can already you already know this but you can already observe that the number of first of all the number of words in english and number of words that are coming out in french it is different so there is no guarantee that the

6:21words number of words need to be equal and also depending on the sentence if the car is at the end of the sentence in english there is no guarantee that that has to be the case in french in this case car is still at the end however there are sentences were european economic zone if you try to translate it into french the equivalent for european will come at the end the zone will come at the end the zone will come at the beginning the French equivalent of zone will come at the beginning so there is no guarantee that the order in which the words appear is exactly one to one mapping there it's not like in language we translate the first word first second word second that's not how it works right so here we

7:01are inputting four words and coming out what is coming out of this sequence to sequence model is three words in French. Now let's see in detail how this works. But before that we also have to understand what happens to these words. So these words are also called as tokens and you might be familiar with that terminology called tokens. It's very popular in the LLM architecture and in many tokenizer schemes one word is not necessarily equal to one token usually one token is a sub word and that really depends on what kind of tokenizer we use but there are many different tokenizers but we are not going to discuss tokenizer right now for all practical purposes for simplicity today we can say that one

7:42word is equal to one token so here you have four tokens in English in the output and in the input and in the output you have three tokens in French okay and what is done internally to represent these words in English or French is you convert those words into vectors and this can be done by passing the words through a neural network so there are there arevec, which are libraries which you can import in Python, which will help you convert word into a given vector. The vector itself can have large number of dimensions. So here I'm just showing you one example or a set of examples where I'm just showing you two dimensions. So this is obviously a very simplified picture, but in reality, the number of dimensionalities

8:25will be 768 or much bigger than that and usually vectors point into a certain direction of space so this is the vector corresponding to apple so this dot is basically the tip of the vector vector is actually something aligned with the direction that's how we represent right but but we can also replace the line uh by just the tip of the vector. So, here these dots are basically the tips of those vectors and you can see that the vector corresponding to apple, orange, mango, etc. are pointing to the similar region of the space. You can say that this is a semantic region in space corresponding to fruits. Similarly, if you had vectors corresponding to cat, dog, cow, etc., they may point to a different region in space, a semantic region in space, which corresponds to, let us say, animals.

9:07In general, it is known that, you might have already seen many examples on the internet, basically, vectors that have similar meaning typically point in the similar direction of the space. And those vectors have very high cosine similarity also. Cosine similarity meaning if you look at the cosine of the angle between these two vectors, the angle is very small and cosine of a small angle is closer to 1. Cosine of 0 is 1, cosine of a small angle is closer to 1. So high degree of cosine similarity these vectors will exhibit. Now how does the neural machine translation work? So I will come back to this figure in a moment. But before I go to this figure, I want to show you another figure how the translation

9:51exactly works. So this is how the neural machine translation or sequence to sequence translation works. There is an encoder and there is a decoder. Encoder is responsible for taking the input from English. So this is the English input part. and the input is passed in the form of these tokens or these words and then the words are converted into corresponding vectors and encoder does something which we will see in a moment that converts this sequence into one single vector called as the context vector. So, this context vector is nothing but it's a vector of let's say 768 dimensions that captures the entire meaning that is present in this whole input. You can already imagine the problem with this if the input sentence was let's say 20 words

10:37or 30 words here it's just four words if the input sentence was longer still the context vector is expected to carry the meaning from those that at the entire long input which is too much pressure on one single vector how can you expect one single vector to carry that much meaning but yet this is how neural machine translation was working and then this context vector is passed as the input into something called as decoder what decoder will do is decoder will sequentially produce tokens in this example in French language and this decoder's final output, final sequence of tokens which forms the final set of output is the English to French translation. Now the question is let us try to slightly in a

11:20bit more depth, let us try to understand what is happening inside this encoder and what is happening inside the decoder. You can already imagine the problem with using this context vector now let's go back so what happens inside the encoder is something like this so you have individual tokens right so this is the token number one token number two number three and number four so you might think that here what is shown in the diagram is different recurrent neural networks assembled in sequence that is not the case these are different you can imagine this to be different time steps here it's not like there are different recurrent neural networks aligned in sequence that is not the case okay it's just that we are representing different time steps in this way so recurrent neural network is

12:08basically a neural network which takes some couple of inputs in this case and it produces one output so what are the inputs that go into the recurrent neural network there are two inputs that go into this recurrent neural network one is the hidden state hidden state here is represented by h you can imagine the hidden state also to be like is represented by h. You can imagine the hidden state also to be like a vector. And another vector that goes into this recurrent neural network is the vector embedding corresponding to your token. So in the first time step, the token is this. That particular token is converted into vector embedding. That vector embedding along with the initial hidden state h0 which you can imagine to be a

12:45random vector initially because there is no token in the beginning random vector this h0 is passed into the recurrent neural network along with the vector embedding corresponding to this and then the output from the RNN is the new hidden state which we call as h1 h1 meaning hidden state after step one now what goes into the same recurrent neural network again so ideally if i want to draw this i should draw it something like this this output is coming out and this output is passed back into you know where this hidden state is passed but since i don't want to do that because that will make the diagram look very cluttered let me just remove that instead of that i'm just showing one more rnn but this is the new time step it's essentially the same rnn now what goes into this rnn again

13:33the the new hidden state h1 which is again a vector plus the vector embedding corresponding to the token is then again it produces the new um new hidden state h2 and this repeats until the final hidden state h4 is produced. Now what is this hidden state h4? That final hidden state after every single token is passed through is called as the context vector. So this whole sentence now h4 carries the information or carries the meaning contained in this whole sentence. Now we have taken a very simple example here, but obviously this sentence could have been much longer in that case instead of h4 it could be h20 or h50 depending on whatever the length of the sentence is, but basically, what it produces is the same, a vector that lives in the same space as that

14:30of whatever the rest of the vectors were. So, if H0, H1, H2, etc. were living in a 768 dimensional space, this final context vector will also live in the same space. Now, the decoder takes the following input. So, as I mentioned already, decoder, the initial input into the decoder, which is also a recurrent neural network based architecture, is the context vector. And this context vector is coming from the encoder final output. So this context vector is the first input into the decoder RNN, which produces the initial output. Initial output is the output vector corresponding to the first french token in this case it is cest i don't know how to pronounce this exactly but this is the first french word that is produced by the rnn then again into the same rnn and by the way this rnn

15:16also produces one more output which is the first hidden state corresponding to the decoder called S1. So these two outputs output vector and S1 is produced from the RNN in the first time step with the context vector as the input. In the next time step actually there are two inputs that go into the same RNN. So again this is not multiple RNNs sequentially aligned or anything these are different time steps of the same recurrent neural network. So in the next time step, we have the hidden state S1 corresponding to the decoder passing into the RNN. Along with that, the vector corresponding to the first French word, actually not the English, not the French token itself, but the vector corresponding to it is passed into the

16:00RNN. And corresponding to these two outputs, sorry to these two inputs, the new two outputs are produced. What are the new two outputs? The first output is the output vector corresponding to the next French word. Second is the new hidden state S2. Like this, the words will be produced in French token after token. And finally to know whether the token sequence production has ended maybe it can rely on an end of sequence token or some special token production but basically this is how decoder converts your context vector into French sentence so now if you look at what is the only input the decoder is depending on to produce the French sentence the only input that this is depending on is the only input the decoder is depending on to produce the french sentence the only input

16:46that this is depending on is the context vector decoder knows nothing else decoder has no idea what sentence was passed into the encoder the encoders passed the sentence that was passed into the encoder could have been very very long the decoder has no idea decoder only looks at this context vector to produce this sequence of French tokens. And that is exactly the problem with this encoder-decoder architecture, which is dependent on the context vector alone in the neural machine translation. So now let us discuss the last part of today's lecture, why this kind of an architecture breaks. Obviously, as the input sentence size increases, you can imagine that more information has to fit in the limited number of dimensions available within the context vector so a vague way to think about this

17:33is imagine that your your context vector dimensionality is 800 okay and let's say in one case in the input you have eight words which means this is not the right thing to say. I know that but what I'm saying is if you have 800 dimensions in your context vector and the number of words you are trying to fit in the context vector is eight, roughly hundred dimensions are available to represent one word. However, imagine that you have 40 words in your sentence. It's let's say it's a very long sentence and same 800 dimensions are still available to, you know, jam pack all of this information in these 40 words. Basically, now you have what? Around 20 dimensions available per word, right? In the first case, you had 100 dimensions available per word.

18:20This is not the right thing to say. It's not like dimensions are assigned to words or anything like that. But if you think of the total number of variables you need to store information, the more you have number of variables, the more information you can store. But the length of the sentence, you are not proportionally increasing the dimensionality of your context vector or anything like that. The context vector remains of constant dimensionality of your context vector or anything like that. The context vector remains constant of constant dimensionality because it is belonging to a certain representation space or in a certain embedding space. So the more the number of words, the more the chances of this whole neural machine translation

18:55via sequence to sequence model breaks. There is also one more thing actually we should think about, which is long range dependency. Let us say we have a sentence like the teacher who was teaching this students smiled. One more thing actually we should think about which is long range dependency. Let's say we have a sentence like the teacher who was teaching the students smiled. So here what may happen is that in the final state which is basically the context vector, the information which came in the beginning because RNN is something which happens sequentially right in the RNN you are passing the first token first English token then the next English token then the next English token so really the information which is there in the beginning part of the sentence especially if it's

19:34a long sentence that information will be lost in the final context vector whereas it will have information more information regarding whatever came towards the end of the sentence. So this is one issue with respect to using this kind of a context vector to pass information from encoder to decoder in the sequence to sequence model. There is one more issue this is the issue which I wanted to mention in the previous slide which is called long range dependency. Look at this sentence the teacher who was teaching a difficult concept to the students smiled. Now, if you really think about this word smiled, who smiled in this case? Is it teacher or students? Actually, it is teacher who smiled. It's not students.

20:21But if you simply look at the geometric proximity or simply look at the proximity between these tokens, the token corresponding to students and token corresponding to smile they are very close to each other so it's very difficult for a language model to understand that in fact the smile is not associated with students smile is associated with the token teacher and look at the positioning of teacher teacher is located at the beginning of this long sentence smile is located at the end of the sentence. So in sequence to sequence models, especially that is based on recurrent neural networks, what happens is this teacher token came at the very beginning in one of the initial steps, but SMILED is coming at the later time step.

20:58So it's very difficult for the translation model to understand that the person who SMILED is actually not students it is teacher this kind of long-range dependency capturing is very difficult using uh rnns rns by the way is not the only architecture which existed uh until attention mechanism came there was also lstm which was an advanced version of i would say advanced modification made to recurrent neural networks however irrespective of that up until attention mechanism came into picture it was very difficult to make sure that long sentences are captured appropriately using context vectors it was also difficult to capture long range dependency in the absence of attention mechanism so there is a famous graph I'm sure you might have

21:45seen this kind of a graph I think on LinkedIn this was I remember seeing a lot of this kind of a graph in 2022 etc on LinkedIn to posted by people but in general the reason why we need attention mechanism which we have not yet discussed like we have we are yet to discuss what exactly is attention mechanism but the reason why attention mechanism kind of shot up in popularity is because it did not matter whether the sequence length increased attention mechanism could still capture the information with long range dependency which means if you are performing a language translation task the translation quality which is plotted here on the y-axis did not decrease with the sequence length which is plotted on x-axis. On the other hand in the neural machine translation in the

22:30sequence to sequence models which depended on one context vector without any attention mechanism you could see that the quality diminishes significantly once the word count in the sentence increases beyond 20. So this was the biggest issue with respect to the traditional before attention mechanism came into picture. So that brings us almost to the end of today's lecture. So I just wanted to say this much before we even discuss about the concepts like attention or badano attention or dot product dot product attention or scaled dot product attention or master you know causal attention etc before we come to all of that i just wanted you to be aware of the fact that this is the reason why you know capturing information using a context vector from long sentences it's very difficult

23:18now in the future also the same terminology called context vector you will come across but you should not imagine that context vector is a bad thing. No, that's not the case. Context vector is an amazing concept. But the way information is captured into the context vector, that is what matters. Here, the way it is captured is not basically using attention mechanism. It is using passing of sequential tokens into recurrent neural networks so there is there is no way for the context vector into context vector to look back and say that if i have a sentence like the cat drank milk because it so let's say this is the sentence so far it's not possible for the sequence to sequence models to look back and say hey here it refers to cat

24:17or here it refers to milk so it's very difficult for that kind of you know preferential attention payment so it's not difficult for that kind of you know preferential attention payment so it's not possible for this mechanism to pay more attention to cat or more attention to milk because here anyway the full sentence is looked at already in the encoder and the whole information is already captured in the context vector now the decoder has to entirely rely on just that one context vector to produce the final output sequence but that's not the case when it comes to attention mechanism which we will look at in the future lecture so in the next lecture we will look at the attention mechanism itself so let's say we are converting this

24:58this sentence the cat sat today into french so the French so the way this translation will follow will be love chat so there will translate to love cat will translate to chat so if if let's say the decoder is currently writing the second token which is chat then the attention has to be paid maximum meaning maximum focus should be on the token cat because chat is directly translation it's a direct translation of cat so here this 0.75 is like an attention score simply how much importance is assigned to that particular token the degree of importance assigned to cat when the decoder is writing the french equivalent of that should be the highest so 0.75 out of 1 that much attention is paid into cat 0.05 is paid to the

25:52very small attention is paid into sat and small attention is paid into today so like this the decoder can actually change these numbers will change depend on which token or which french word the decoder is writing at in this moment and that made the attention mechanism so special but that we will discuss in much more detail in the next lecture so allowing the decoder to look back completely change the game and we will start with badano attention which was proposed in 2014 and by the way badano attention is also called as additive attention. So we will see why it is called additive. And after that came the dot product attention. And after that came the, so dot product attention came in 2015.

26:34And after that came the scaled dot product attention in 2017, which was proposed in the attention is all you need paper. So we will discuss these things in the subsequent lectures. So thank you so much. See you soon in the next lecture.