The attention family · Vizuara · Transcript & summary by WhipScribe
It derives the additive score formula step by step from a worked English-to-French example, which makes the later jump to dot-product and self-attention easy to follow.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“Which is too much pressure on one single vector.” Instructor · at 1:26 — open in the transcript →
“Vector v is actually a learned parameter; it is not something which we define.” Instructor · at 33:58 — open in the transcript →
Lecture 3 of Vizuara's nine-part attention series is about the mechanism from a 2014 paper whose first author is Bahdanau, usually called additive attention. Its target is the weakness of RNN encoder-decoder translation covered in the previous lecture: one fixed context vector had to hold an entire English sentence, and quality collapsed once inputs passed roughly 20 words. In the instructor's seven-bucket map of attention variants, this lecture sits squarely in the first bucket, scoring.
The fix is a context vector rebuilt for every output token. A small neural network compares the previous decoder state with each encoder hidden state and returns a raw score; softmax turns those scores into weights that sum to one; the weighted sum of the encoder states becomes that step's context. While producing the French word for cat, the encoder state for cat gets most of the weight. The name comes from the scoring function itself, which adds a projected decoder state to a projected encoder state, applies tanh, and takes a dot product with a learned vector v to reduce the result to a single number.
The instructor also follows a full decoding step: context and decoder state are concatenated into 1,536 dimensions, projected onto a 50,000-token vocabulary and sampled greedily or by top-k. Attention weight matrices are presented as a lasting diagnostic tool, readable row by row, and a European Economic Area example shows strong weights running off the diagonal when French reorders words. Differing sentence lengths are handled by the end-of-sequence token. The lecture closes by pointing to Luong's 2015 dot-product scoring and the scaled version in Attention Is All You Need.
The Vizuara instructor explains that each score comes from a tiny neural network that takes the previous decoder state s and one encoder hidden state h. It multiplies s by a weight matrix W1 and h by W2, adds the two projected vectors, applies tanh for non-linearity, then takes a dot product with a learned vector v to collapse the result into a single number: score = v · tanh(W1·s + W2·h).
According to Vizuara's Lecture 3, the name comes from the scoring function: the weighted decoder state (W1·s) and the weighted encoder state (W2·h) are summed before tanh and the dot product with v. That addition of the two projected states is what makes it additive, in contrast with the dot-product scoring that Luong proposed in 2015.
Vizuara's instructor says RNN encoder-decoder models, used until around 2014, squeezed a whole source sentence into one final hidden state used as a constant context vector for every output word, and translation broke down beyond roughly 20 words. Bahdanau attention replaced that with a context vector recomputed at each decoding step (C1, C2, C3 and so on), so the decoder gets a fresh mixture of encoder states focused on what it needs at that moment.
In Vizuara's lecture, attention scores are the raw numbers produced by the scoring network for each encoder state, and they need not sum to one. Passing them through softmax (e to the x_i divided by the sum of e to the x_j) turns them into attention weights that form a probability distribution, and those weights multiply the encoder hidden states to build the context vector.
The Vizuara instructor reads the matrix row by row: each row belongs to one output word and its cells, which sum to one, show how much that word attends to each input word. The brightest cells reveal which words are translations of each other. The matrix is square only when input and output lengths match; cross-sentence attention is generally rectangular, unlike self-attention, which is always square.
In Vizuara's walkthrough, the decoder concatenates the current context vector with its previous state; with 768-dimensional vectors that gives 1,536 dimensions. A linear layer projects this onto the vocabulary, for example 50,000 tokens, and softmax turns the logits into probabilities that are sampled greedily or with top-k. Another layer produces the next decoder state, and the loop runs until an end-of-sequence token appears.
The Vizuara instructor says additive (Bahdanau) attention computes each score with a small neural network, while Luong's 2015 paper showed the score can come from a plain dot product with no network at all. Scaled dot-product attention, introduced in Attention Is All You Need, became the scoring method used by nearly every later attention mechanism, and both are the subject of Lecture 4.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Hi, welcome back to a new lecture from the attention family lecture series. Today's lecture is about additive attention. However, this attention is more famously known in the name of Badano attention. This was actually introduced a while ago in 2014. And it's very important for you to understand Badano attention if you really want to appreciate the attention mechanism, the self-attention mechanism that we are eventually going to discuss in the upcoming lectures. So, today's lecture is purely about Badano attention or additive attention. A quick recap of what we discussed in the last lecture. In the last lecture, we looked at neural machine translation through sequence to sequence models. The model was actually having an encoder and decoder architecture. The encoder will
0:45have input as the English sentence and decoder will have an output in the form of French sentence. So English is the input which is getting translated into French and we also discuss the biggest shortcoming of this architecture which is the context vector. The context vector as an idea is an amazing idea. The only problem here is that this context vector is one single context vector, which is expected to capture the full information that is there in this sentence, irrespective of the length of this sentence, whether it is four words or four tokens, or whether it is 20 words, it doesn't matter. The context vector is expected to capture all of this information, which is too much pressure on one single vector.
1:28And that is exactly the reason why this kind of architecture failed when the length of the sentence exceeded, let's say, you know, above a threshold of 20 tokens or 20 words rather. So in the last lecture, we discussed how this encoder decoder architecture worked based on recurrent neural networks and this kind of architecture existed till around 2014 and the recurrent neural network converted the inputs into a hidden state and the final hidden state from the encoder is the context vector, the context vector goes as an input into the decoder and the decoder will one by one produce the output tokens. Now, in the first lecture, we discussed that whenever we talk about an attention mechanism, we will not be just learning about the mechanism, we will learn where to fit it in our bucketing
2:18because if there are 50 attention mechanisms, all of them can be nicely fit into few different finite number of buckets. Even if in the future, a new attention mechanism comes, we will be able to fit in one of these seven buckets. And this figure I had shown you in the first lecture where there are seven buckets. First bucket is about scoring. Scoring meaning how the attention score is calculated. So, in today's lecture, we will discuss primarily about the attention score calculation. We will not be discussing any other aspects. The second bucket is connectivity, meaning which tokens are allowed to attend which. We will see that in the upcoming lecture about the self-attention. Then third is about number of attention heads and the KV cache.
3:00Fourth is about the complexity of the attention calculation, the attention weights calculation. Usually, it is quadratic, but there are architectures which make it sub quadratic. Fifth is about how the attention mechanism is implemented in GPU. For example, flash attention is a great example for this and how position injection position information is injected into the attention mechanism. That is the bucket number number seven bucket number six and bucket number seven is what is the domain in which we are implementing this attention mechanism whether it is text video audio image or whether it is graph so in today's lecture whatever attention mechanism we are discussing the additive attention or badano attention is coming under this bucket of
3:41scoring it does not mean that there are other mechanisms which does not fall in this bucket. That is not the case. There are other mechanisms which will maybe compute attention mechanism slightly differently, which we will discuss in the next lecture. So, that next lecture also most probably fall under this scoring bucket. And later when we learn newer and newer attention mechanisms, we will see how the rest of the buckets are also coming into picture. But ultimately, irrespective of the number of attention mechanisms that we are going to discuss in this lecture series, all of them can be nicely fit into one or few of these buckets. So today's lecture, Badano attention and next lecture will be about dot product attention
4:23followed by scaled dot product attention so i'm hoping that these two attention mechanisms i can discuss in the next lecture so we will divide today's lecture into three parts in the first part we will see the idea behind badano attention which is literally the attention mechanism and why is it fixing something which could not be fixed in the RNN based sequence to sequence models and we will also see the full architecture of Vadanav in the second part of our second section of today's lecture and in the last part we will also discuss how we convert the attention scores into attention weights using softmax and we will also see why vadanova attention is actually called as additive attention that we will see in the last section it's actually very simple but i will show
5:11you exactly the architectural reason why it is called as additive attention okay so firstly the core idea behind vadanova attention the core idea is this whenever decoder is producing a new token, do not rely on a single context vector. When I say single context vector, I mean the context vector that was produced in the encoder decoder architecture in the sequence to sequence model was a single context vector. It was not changing. That context vector alone was responsible for producing every single token in the output. Here on the other hand, the context vector keeps changing depending on what token we are producing. For example, if we are producing the French equivalent of cat, so this chat is basically, it's not not english word chat it's like the french word
6:05for cat so let's say we are translating this sentence the cat sat today into french and let's say currently we are decoding we are basically producing in the output the word in the french word corresponding to cat which is chat if that is the word that is being produced, the degree of attention or degree of focus that we have to pay to the incoming tokens or incoming hidden states should be such that cat should get the highest attention because now cat is being directly translated. Of course, translation is not a word by word thing directly, but sometimes it is. So in this case, if we were to say how much attention we have to pay or how much focus we have to pay to every single incoming input token or every single encoder states, we should
6:52say most of the attention should be paid to the hidden state corresponding to cat and remaining which should be paid to the rest of the tokens or to the rest of the hidden states we will see the exact architecture so don't worry about that yet this is the idea behind um devising something called as attention scores we will also see how attention scores are calculated but these are the attention scores so this is the attention score between the and chat. This is the attention score between sat and chat and so on. So now you might be thinking, what do these attention scores do? So before we try to understand what are we going to do with these attention scores, let's take 30 seconds to recap how the architecture of the encoder looked like in the last lecture.
7:43In the last lecture, we discussed regarding the recurrent neural network based architecture for sequence to sequence models for neural machine translation and what we had was something like this so if you had an english sentence this is a car each of those english tokens are passed into the recurrent neural network along with so initially along with the initial hidden state this english word is converted into a vector embedding so along with this hidden state and this vector embedding rnn will produce the next hidden state which is passed again into the same rnn here we are not displaying multiple rnn's in series here instead these are different time steps this we had discussed in the last lecture so this new hidden state along with the embedding corresponding to is is passed into the rnn and
8:29finally we have the context vector so once this context vector is produced it is constant it's not going to change irrespective of what token we are producing in the decoder this same context vector was responsible for producing the entire output sequence but in the decoder this same context vector was responsible for producing the entire output sequence but in the case of attention scores what we actually do is the following what we do is we have the attention scores which will vary depending on what token we are producing next so if we are producing the french equivalent of cat which is chat the attention score will be highest corresponding to the token cat so now what the decoder receives will be a context vector which will depend on this
9:16current attention scores the context vector will be a weighted sum of the vector embeddings of all of these tokens. So the will be converted into a vector embedding. So this is the vector embedding corresponding to the that will be multiplied with 0.05. This is the vector embedding of cat. That vector will be multiplied with 0.75 and all of these vectors which are weighted are added. And what are these weights these weights are nothing but the attention scores that we calculated technically a better term to say this instead of attention scores is to call it as attention weights because these numbers essentially add up to 1.75 plus 0.15 is 0.9 0.9 plus 0.05 plus 0.05 they will add up to one so now the difference between what was done
10:08before badano attention versus what happens within the badano attention is that these weights changes depending on what token we are producing and based on that the resulting context vector which is a weighted sum of these embeddings that will also change so that is the key difference that you have to remember all right so that is what i have written here in this this one sentence just pay attention to this earlier the decoder got the same squeezed vector at every step what is that same squeeze vector it is the context vector now so now meaning when we have attention mechanism it it meaning decoder decoder gets a fresh mixture at every step focused on whatever it currently needs so if the decoder is producing
10:51the equivalent of cat right now the context vector will look different versus if the if the decoder is producing the equivalent of today the context vector will look something else. So this diagram should once and for all make it very very clear for you forever. So before attention mechanism you just had one context vector which was agnostic to what token you are producing at the output, output meaning at the decoder whereas with attention you have different context vectors. So, C1 will be the context vector that we will have when you are producing the first token at the decoder output. C2 will be used when you are producing the second token at the decoder output and so on. So, this is the major difference
11:36that I want you to remember what the attention mechanism managed to do and this has profound effects. This is not something that is too simple. You have to understand that this has profound effects this is not something that is too simple you have to understand that this has very far-reaching consequences so now what is attention attention is basically how much focus you are paying to each token or each vector embedding in a given moment and these attention scores can be converted into attention weights so scores are raw numbers but they can be converted into attention weights which will add up to one so it will look like a probability distribution by passing the attention weights through something called as softmax function so we will discuss about softmax
12:23function soon but basically scores are passed through softmax function to produce attention weights so which means when you are multiplying the individual embeddings with the with their corresponding weights and when you are adding those embeddings together essentially you are doing a weighted average or i would like to call it as a weighted sum okay so now now that we are clear what is attention, that definition is out of the way, let's look at how the Badano attention mechanism worked. So we have to discuss that in part two. And in part three, we will also discuss why Badano attention is called as additive attention. What is additive about Badano attention? Let's see that also. So in Badano attention, this is what happens.
13:05You have an encoder, I have one more diagram which shows the full detail of Badano attention mechanism but this is a simplified version. You have the encoder over here and you have the decoder over here. Now let's say the current state of the decoder is such that the word le, le is the translation for the, the french translation for the let's say that is already written now next we are trying to write the second french token which may be the translation for cat but obviously translation does not work like that it's not like you translate the first word then the next word then the next word language does not work like that but in this specific example yes we are translating cat so now what is the context vector the context vector is c2 why is
13:50it c2 and not c1 because the first token is already produced using c1 first token lay was produced when we had c1 but now we have c2 and how is c2 produced to produce C2 you make use of h1 h2 h3 h4 what are these these are the hidden states basically these are hidden states produced by the recurrent neural network by utilizing the these inputs so these inputs are converted to corresponding embeddings that embedding along with the output of the RNN from the previous step is utilized for producing these series of H and what is this H? H is the hidden state. This is hidden state 1, H2, H3 etc and these are also vectors living in a certain multi-dimensional space. Now how do you get context vector out
14:39using these H1, H2, H3, H4? The context vector C 2 is 0.05 times h 1 plus 0.75 times h 2 plus 0.15 times h 3 and plus 0.05 times h 4. So, it is the weighted average of these hidden states. Now, this context vector is now the only input that goes into the decoder. The decoder also has RNN architecture and this time this RNN is having two inputs. One input is the decoder state S1. This decoder state S1 was produced by the same RNN in the previous step. In the previous step where it produced late, this hidden state or the decoder state S1 was produced. So along with S1, this context vector c2 is also passed into the rnn and this rnn produces two outputs one is the next french token or next french word now this
15:34this is not english word chat this is chat i'm not sure if i'm pronouncing it correctly but this is the french translation for cat so the french translation for cat is produced by the RNN plus the next hidden state or next Decoder state s2 is also produced Now you can imagine what will happen now in the next step you are supposed to the output decoder at the output is supposed to produce s3 using s2 and c3 and How will c3 be produced c3 will be again the weighted sum of h1 h2 h3 and h4 however this time what will be different these weights will be different the weights will depend on uh what is the next token that you are going to produce okay so i'm not showing you visually that detail here hopefully in the next diagram it will be clear
16:22for you but this is this is the true essence of badano mechanism these attention scores which are varying dependent on which token is being produced is the core essence of badano attention now this is a slightly more detailed uh view of badano attention and let's let me show you this time how the scores are influenced by the decoder hidden states also. Because here in the previous figure, you might have a question. Okay, these scores, how exactly are these scores computed? What are the inputs that go into the computation of these scores? Let's actually have a look at it. So this diagram, although it looks a little bit involved and maybe a bit intimidating, don't worry, it's actually very simple. I'll walk'll walk you through this again these are the inputs in english the cat sat today and we
17:11have to produce the french equivalent now let's say already the first token is produced so lay is produced already it's already written now we are about to produce the next french token which is corresponding to cat so it is chatat so first the cat sat today is converted into hidden states h1 h2 h3 and h4 who is doing this conversion the recurrent neural network is doing the conversion that you already saw in the previous diagram now somehow uh h1 h2 h3 and h4 along with the decoder state s1 somehow we need to convert into attention scores okay. So, attention scores are computed using these functions f. What are these functions actually? These functions are simple neural networks. I will show you the architecture of
17:59this neural network it is extremely simple, but basically there are these tiny neural networks which will, so this F which is a neural network is taking H1 and S1 as an input and with these two as an input, it will produce the attention score corresponding to the encoder state H1. Similarly, this neural network, it is the same architecture, it will take x2 and s1 as an input and with those two things as the input it will produce the corresponding attention score like that four attention scores are produced now please note these are attention scores these are not attention weights these are raw numbers which will not add up to one now you can convert these numbers which will not add up to 1 into a set of numbers that
18:46will add up to 1 like a probability distribution by passing it through softmax. Softmax I hope you know the softmax function, softmax function is nothing but e raise to x i divided by sigma e raise to x i. So, this is the softmax function. So, if you pass through this softmax function, you can convert the numbers into corresponding attention weights. And then you take the, you calculate the context vector c 2 by taking the weighted average of the, or weighted sum of these encoder states so h1 multiplied by 0.05 plus h2 multiplied by 0.75 plus h3 multiplied by 0.15 so on and that will produce c2 now using c2 and s1 the recurrent neural network can produce so using these two things as the input the recurrent neural network can produce two outputs one is the c3 which is the next uh sorry which one is s2 which is the next state
19:52and second is what the next token should be so one way to do that is you can uh kind of concatenate c2 so c2 is a vector right c2 is a vector why because these encoder hidden states are vectors let's say the dimensionality is 768 dimensional space. So you have C2, which is the context vector and S1 is nothing but the decoder current decoder state vector. So these two vectors are concatenated and then they can be passed through a particular linear layer. Linear layer is nothing but a layer which will have so let's say this is 768 comma 768 so total dimensionality 768 multiplied by 2 whatever that number is is it is it 1 5 2 6 um let me do that 768 times 2 no 1 not 1 5 2 6 1 4 5 2 my bad yeah anyways so 1452 how is it 1452 768 multiplied by 2 right I typed wrong is 1536, let me just erase this.
21:18Now you want to go from something which is of 1536 dimensions into what? You want to go from that many dimensions to whatever is your french vocabulary size so yes let's say your french vocabulary size is 50k basically token size not necessarily french vocabulary size whatever is your vocabulary size so if it is 50k you are having now a linear projection layer which will project from 15 1536 dimensions to 50 000 dimensions okay and while doing that projection you will also pass through a softmax from 1536 dimensions to 50,000 dimensions. And while doing that projection, you will also pass through a softmax function so that 50K raw logits you can convert into a probability distribution over 50,000 numbers.
21:57Then among that, whichever has the highest probability you can pick, it really depends on the sampling techniques. Sometimes there are greedy techniques, which will pick whichever has the highest probability sometimes there are top top k techniques where you will be sampling from top k items but essentially this is how the next token will be produced and you can also have one more layer in the recurrent neural network which will produce the next state which is s2 that s22 will be another vector. So, in this case since we are talking about vectors of dimension 768, this S2 will be a 768 dimensional vector. Then that S2 will basically take place over here and then S2 and H1 will produce the next core,
22:41S2 and H2 will produce this core, H2 and s3 will produce this core like that and those cores will be used for multiplying and adding these h1 h2 s3 s4 to produce c3 and c3 and s2 will be used for producing the c4 not c4 the next token and s3 and like that this process will continue until end of sequence or some special sequence special token is arrived at so this is in a nutshell how Badano attention is working and the core part of this Badano attention mechanism in comparison to what was happening before Badano attention is this attention score plus attention weights this part. This is the core part of rather I should draw a slightly bigger rectangle. So, this plus the context vector because the context vector was there before, but this time the context vector is very dynamic. So,
23:28this whatever is there in this red color rectangle that is the core contribution of what the attention mechanism could do. So, here I am just once again mentioning the same thing. So, here S 0 is the current state of the decoder and to decide s 1 to decide the next state the current token and the current context vector is used to decide the state after that we have a new context vector c 2. So, this is the only thing which I want you to notice the context vector c 1 c 2 c 3 etc are different context vectors they are not the same context vectors this i have already discussed in detail so i'll be skipping this and yeah this also i have discussed but just one one thing to notice is currently if the next token
24:18that is being written in french is the equivalent for cat which is shat the attention attention scores or attention weights might be distributed in such a way that in this moment cat may have the highest attention score basically the attention score between shat and cat will be the highest on the other hand if we are writing this neuer neuer is the equivalent the translation for black the attention score between neuer and black will be the highest the attention score will be neuer and cat Neuer and the they will be low so see the attention scores are dependent on what is your current decoder state okay this also I have already mentioned basically this slide is trying to ask the question what happens if the number of words in your input sentence which is English and
25:03number of words in output which is French are different and of course they can be different there is no guarantee that translation results in the same number of words but that's not a problem because in the decoder state as we are producing new tokens at some point the decoder may produce the end of sequence token at that point you will you will say that your translation is complete so the point at which that at which that happens need not necessarily be after exact same number of tokens are produced at the output. Maybe the output has produced lesser number of tokens than the input. Maybe the output has produced more number of tokens than the input. It depends on the language, what exact sentence are we trying to translate, et cetera.
25:42So finally, this matrix called attention weights matrix is something which we are forever going to see as long as you are working with transformer architecture. Throughout this series we will be seeing this kind of a matrix. In this case it is a it happened to be a square matrix, but there is no guarantee that we will be looking at a square matrix because these are the attention scores between or attention weights between two different sequences so please note in this case it's technically not self-attention we have not discussed self-attention yet it is still in the upcoming lecture but self-attention means you are attending one sentence is attending to itself which means the attention weights matrix will be a square matrix the number
26:25of tokens over here and number of tokens over here will exactly be the same if it was self-attention in this in this case technically it is not self-attention because a sentence in English is attending is paying attention to a sentence in French in specific case, the number of words in the input in English and number of words in French output happen to be the same because of which you are seeing a square matrix. Otherwise, this would have become a rectangular matrix. Now, pay attention to individual rows. So, these rows are showing the attention scores. So, this 0.8 comma 0.1 comma 0.1 will add up to 1. those are the attention scores between lay and the lay and black and lay and cat and similarly this row they will also add up to one
27:13because those are the attention scores between shat and the shat and black and sat and cat and obviously the attention score between shat and cat should be the highest because chat is the direct translation of cat so you can make a lot of sense by looking at which of these cells in the attention weights matrix is the dominant ones because those are basically the tokens that are paying most attention to each other all right and this kind of insight you will also be able to derive later when we look at self-attention between a set of tokens in a sentence with its own set with its own tokens okay and this is another example where you are looking at a sentence the european economic area that's the english sentence and la zone economic
28:01european this is the corresponding french obviously my french pronunciation is not pronunciation is not correct but uh one thing you can notice is that it's not like this attention weights matrix is having dark colors or you know prominent values across the main diagonal there is no guarantee that would have happened if languages were translated in such a way that one word is translated first, second word is translated next. If the language translation was working like that, then yes, most probably in this case, this main diagonal elements all would have been dominant. But here you can see that here it's, I would say in the transverse direction of the direct diagonal. That's because here area is translated to zone, but zone is
28:43coming in the second position. Similarly, European is translated to European, but that's coming at the last. So if you look at the attention scores, yes, the attention score between European in French and European in English, they should be highest, which is the reason why the 0.83 is highest. Similarly, zone and area are translations of each other which means this cell which captures the attention score between those two tokens it is the highest right so this is how attention map visualizes how your attention scores are laid out now the last part how exactly are we computing attention scores within the neural network so earlier i showed you a function f that f was taking two inputs one was the hidden state h and one was that the hidden state h from
29:31the encoder and one was the decoder state called s so from these two inputs how exactly are we calculating the attention weights attention scores let's look at that and let's also discuss why the badana attention mechanism is called as additive attention so firstly the way we go from attention scores to attention weights we have already discussed this the scores let's say they are 2 4 1 3 these are raw attention scores we have to we have to convert this into probability distribution called attention weights by passing through a softmax function. So obviously whichever number was dominant before, that number will dominate after even after taking softmax. But there are certain nuances here like the way you take softmax, there is a normalization
30:17that is typically done, but none of those techniques were introduced in the Badano attention paper. By the way, the Badano attention paper is very famous. It has around 45,000 citations, if I remember correctly, 44,000. So this is the paper. And this came out in 2014. And the surname of the first author is Badano. So that's how this name came. He's like a very prolific researcher researcher you can check the Google Scholar more than hundred thousand citations like a very big name in the modern artificial intelligence so this is the first step which is the score converted into weights the other step which I called which which the additive attention part is coming in the function F so earlier this whole setup whatever is there in this red color rectangle I had called as
31:09a function f. So, I told you that when I have a function f there are two inputs the hidden state from the encoder and hidden state from the decoder right. So, encoder state is h 2 decoder state is s1 these are both kind of vector embeddings and both will go as the input into this function f and somehow the score is coming out and how is it happening it's happening in this way so first w1 multiplied by s1 so w1 s1 obviously whether it's w1 s1 or s1 w1 depend will depend on the dimensionality w1 s1 so we are we are multiplying with a weight so weights multiplied by s1 plus w2 h2 okay so this is addition so this addition is happening and that's the reason why we are calling badano attention as additive attention and then you pass
32:02this through a tan hyperbolic activation function to bring non-linearity. Otherwise, whatever you do here will be linear additions. But neural networks, if neural networks need to be helpful as a universal function approximator, if neural network wants to be able to approximate any kind of pattern, there should be non-linearity in the form of activation function it can be tan hyperbolic it can be sigmoid or raylu but tan hyperbolic is applied on top of so i hope you know the tan hyperbolic graph it basically runs from minus one to one on y-axis and x-axis inputs can be anything so like something like this sigmoid will run from 0 to 1 so this is the tan hyperbolic graph and then there is one step called doing a dot product with a vector v so what is happening here is so
32:58when you do tan hyperbolic of w1 s1 plus w2 h2 here basically these are vector additions so s1 lives in a certain space h2 lives in same dimensional space so you are doing or if that is not the case the w1 will basically project it from one space to another so w1 will project s1 from its current space to another dimensional space w2 will do the same but ultimately the dimensionality of the resulting spaces after w1 s1 multiplication and w2 h2 multiplication will be the same that's the reason why you are able to add them element wise so whatever is resulting will be like a vector now what do we want at the output of this f at the output of f i want one number that number should be the attention score between h2 and s1 right so if
33:43i want one number one number means it's a scalar so somehow this this product which results after time hyperbolic operation which is a vector i want to convert into a scalar so for that you do a dot product with a v with a vector v and this vector v is actually a learned parameter it is not something which we define it the vector v emerges or it evolves as a process of as a part of training of this function f so when you do the dot product the result is a scalar and that scalar is the attention score and then the attention scores you get from get between h1 and s1 h2 and s1 h3 and s 1 and H 4 and S 1 those attention scores you pass through softmax to get attention weights. So, this is the this is the additive part and that
34:30is the reason why we are calling Badano attention as additive attention. So, that is all for today's lecture let us do a quick recap. So, what we learned today is about additive attention it is additive because the final attention score is v dot product with tan hyperbolic of w 1 s plus w 2 h. So, this weighted decoder state plus weighted encoder state, the sum of those two that is that the fact that we are doing that sum is the reason why this is called as additive attention and the attention scores are converted into attention ways. And what we did not discuss today is dot product attention and scaled dot product attention. We also have a few more details to discuss regarding why the scaling works etcetera in
35:13terms of from the perspective of softmax that we will discuss in the next lecture. But if we were to look back at this particular bucket, so this bucket we looked at in the last two lectures. So, today we focused on this scoring part, how the attention score is computed. We compute attention score using a neural network. But in the next lecture, we will see how attention score can be easily computed without any neural network with a simple dot product and the logic behind that dot product. So, we are still in the bucket number 1, it does not mean that future mechanisms do not use the bucket number 1, of course it uses bucket number 1, every future mechanisms that we will learn, every future attention mechanisms that we will learn will use the attention
35:58score calculation which is predominantly scaled dot product attention which we will discuss in the next lecture. But please understand that currently we are yet you know still focused on this particular bucket in this lecture as well as in the next lecture ok. So, in the next lectures what we will be doing is we will look at how the field evolved from additive attention to dot product attention which was proposed in this Liuong paper in 2015. This paper has around 11000 or 12000 citations pretty famous paper and then probably the most famous paper in the history of artificial intelligence the attention is all unit paper where they implemented scaled dot product attention. So we will discuss about dot product attention and scaled dot product attention in the next
36:42lecture after that we will be fully ready to move towards self-attention mechanism so i hope you are enjoying the series thank you so much for listening we will meet again in the next lecture take care until then bye
← Part 2Before Attention: The bottleneck of a single context vector | The Attention Family, Lecture 2Part 4 →Dot product attention and scaled dot product attention | The attention family series - Lecture 4