The attention family · Vizuara · Transcript & summary by WhipScribe
Thirty minutes of worked numbers that make the square root of d_k in the transformer attention formula obvious rather than memorised.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“There are no learnable parameters. It's a simple dot product between these two vectors.” Instructor · at 6:14 — open in the transcript →
“When you do not scale it takes an approach called as winner takes all, meaning whichever is dominant, whichever is winner, completely starts dominating.” Instructor · at 27:59 — open in the transcript →
The Vizuara instructor uses Lecture 4 of The attention family to close out the scoring stage of attention, the step that decides how strongly two token vectors relate. Two scoring rules are covered: plain dot product attention, introduced by Luong and co-authors in 2015, and the scaled version from the Attention Is All You Need paper. Both are set against the additive Bahdanau scorer from the previous lecture, which ran a small neural network with learned weights and a tanh just to produce one number.
The case for the dot product is geometric. Tokens live as vectors in a high-dimensional semantic space, related words such as animals or fruits point in similar directions, and the dot product is large when the angle between two vectors is small, zero at 90 degrees and negative when they point apart. That makes it a free, parameter-less similarity score. The catch is size: for random vectors the score grows with the square root of the dimension, so at hundreds or thousands of dimensions the raw numbers become large and softmax turns small gaps into near one-hot weights.
The instructor demonstrates this in a Google Colab notebook and an interactive lab, comparing softmax weights before and after dividing by the square root of d as the dimension climbs from 1 to 4,096. The training argument is that a saturated softmax makes an uncertain guess look confident, which shrinks the loss and the gradients the model needs early in training. Scaling restores sensible distributions, which is why the scaled dot product is the scoring rule nearly every later attention mechanism in the series uses, starting with self-attention in Lecture 5.
The Vizuara instructor explains that the dot product of two randomly initialised query and key vectors grows in magnitude with the square root of their dimension. At the hundreds or thousands of dimensions real models use, raw scores get large enough that softmax saturates and hands almost all the weight to one token. Dividing every score by the square root of d cancels that growth, so the scores stay in a similar range whatever the dimension and the softmax weights stay meaningful.
In Vizuara's Lecture 4, additive (Bahdanau) attention scores a pair of hidden states with a small neural network: each is multiplied by its own weight matrix, the results are added, passed through tanh and then dotted with a learned vector v. Luong's dot product attention, published in 2015 a year after Bahdanau's paper, drops all of that and simply takes the dot product of the two vectors. It has no learnable parameters at all, and the instructor argues a scoring step does not need to be more complex than that.
The Vizuara instructor's reasoning is geometric. Tokens are vectors in one shared high-dimensional semantic space, and related words, such as cat, dog and tiger, or apple, orange and pineapple, point in similar directions. The dot product tracks the angle between two vectors: it is a high positive number when they point the same way, exactly zero at 90 degrees and a large negative number at 180 degrees. Tokens that are semantically related therefore get high scores, so the dot product can be used directly as the attention score.
In the Vizuara instructor's Colab notebook, a query scored against two random keys at 1,000 dimensions produced raw scores of 252 and 258. Softmax on those gave about 0.2% and 99.7%, while dividing by the square root of 1,000 (about 31) first gave roughly 45% and 54%. At 4,096 dimensions, raw scores of 1,032 and 1,042 became about 0.003% and 99.996% unscaled, but 16.13 and 16.29 after dividing by 64, which softmax turned into roughly 46% and 54%.
The Vizuara instructor argues that a model in early training needs its predictions to be unconfident, because a less confident correct prediction produces a higher loss, larger gradients and therefore real weight updates. Unscaled scores break this: near-equal raw scores of 754, 728, 741 and 739 came out of softmax as about 99.999% for the first and close to zero for the rest. The model looks certain when it is not, so the loss shrinks and learning stalls. After scaling, the same scores gave about 31%, 19%, 24% and 24%.
Vizuara's Lecture 4 credits scaled dot product attention to the Attention Is All You Need paper, which the instructor says has almost 260,000 citations, and plain dot product attention to Luong and co-authors in 2015. The instructor calls the scaled version by far the most popular and most frequently used scoring rule, and says nearly every attention mechanism covered later in the nine-lecture series uses it, starting with self-attention, which Lecture 5 builds by hand from scratch.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:01Hi and welcome back to a new lecture from the attention family series. In this lecture series we are looking at various attention mechanisms starting by building the basic foundations and going all the way till more modern and more complex attention mechanisms including Kimi, Delta attention, etc. Today's lecture is about two attention mechanisms dot product attention and scaled dot product attention. In the last lecture we looked at Badano attention and before that we discussed regarding what was existing before even attention mechanism came into picture. So today's lecture is the last lecture before we enter into self-attention mechanism. So let's do a quick recap of what we covered in the last
0:43lecture. In the last lecture we discussed what exactly is attention score and attention weights and how these numbers help us compute context vectors that are dynamic. So, before attention mechanism came into existence, the encoder produced a context vector and that context vector was fixed and decoder always had access to just that one context vector but that is not the case after attention mechanism after attention mechanism we have these attention weights that can be used for computing new context vectors depending on what token the decoder is producing at the moment so we discussed about badano attention in detail in the last lecture and now and we also discussed why it is called as additive attention mechanism because we are multiplying
1:30by a weighting factor and that weights are multiplied with the hidden state from the encoder and hidden state from the decoder and those two things are added up when they are passed through an activation function and then you take a dot product with a learned vector and that dot product produces the attention score. So I am just recapping what we discussed in the last lecture before we move ahead. So once again we are back to this diagram which we will be again and again we coming back to. So first category or the first bucket is where we are still at which is the scoring bucket. Scoring bucket meaning whether this attention mechanism is looking at how the attention score calculation happens or not.
2:13So, in the last lecture we were still in the scoring bucket because the attention score calculation in the additive attention mechanism is different from what we will be discussing today. Today we will be looking at two attention score calculations. One is the dot product attention score and another is the scaled dot product attention score. Let us see how that works. This is again very straightforward. So if you are very familiar with attention score calculation from a foundational perspective, you can completely feel free to skip this lecture and you can move ahead with the next lecture. But otherwise, if you want a good refres refresher please feel free to stay on. So the two attention mechanisms we are discussing today the first one the dot product attention
2:51was proposed in this paper by Luong and co-authors this paper came out one year after the Badano attention paper so this was in 2015 and this paper currently has around 11,000 or 12,500 citations. So this is the first paper that is the paper that is behind the dot product attention mechanism. The second paper which is behind the scaled dot product attention mechanism is none other than the attention is all unit paper which now has almost 260,000 citations. So let's go through this one by one. So today's lecture has three parts. In the first part, we will discuss about the dot product attention mechanism and why is it simple and why does it even make sense compared to the additive attention mechanism proposed in Badano's paper. In Badano's paper, actually that attention mechanism
3:41included a neural network itself, which is a bit complicated, right? If you think of the fact that you need a neural network to just compute a simple attention score it is a little bit complicated. So, we will discuss why attention score calculation need not be that complex a simple dot product is enough. Second is we will also discuss the problem with respect to increasing dimensionality. So, if the dot product between the two vectors which is resulting in the attention score calculation if the dimensionality of those two vectors increase there is a chance that the softmax scores which are resulting from the output finally the softmax course which are corresponding to the attention weights can saturate so we will see that issue
4:23we will visualize it i have have a Google collab notebook, which I want to run and I want to show you some things. And in the last part of the lecture, we will discuss regarding how this problem of dimensionality increase resulting in saturated softmax scores can be fixed by scaling the attention scores. So these are the three parts. So first let us discuss about the dot product. The idea here is in Luong's paper, the question that was asked was what if instead of having a neural network based function for calculating attention scores, why do not we simply use a dot product? So, in the last lecture, we saw that this neural network which was like f that was performing a complex function like this it was
5:05doing tan hyperbolic of w1 q so instead of q it was the hidden state from the encoder plus w2 multiplied by hidden state from the decoder multiplied by or dot product with a learned vector v so this was the function f which was essentially this small neural network but look at how complex this is and here query q and k are kind of like universal notations for query and key usually you take dot product between query and key the definition of what is query what is key what is value etc we will discuss in detail in the upcoming lectures. For the simplicity purpose for today's lecture, just imagine that query is corresponding to let us say hidden state from encoder, key is corresponding to hidden state from decoder
5:55or you can simply imagine them to be vectors corresponding to two different tokens that are trying to attend to each other. So you have two tokens converted into vectors, you want to calculate what is the attention score between those tokens. Okay. As simple as that. In additive attention mechanism, this whole thing is a little bit complex. In dot product attention, it is extremely simple. There are no learnable parameters. It's a simple dot product between these two vectors. Okay. Now let's look at the logic behind taking a dot product. The logic behind taking a dot product is fairly straightforward. So first of all, the vectors that we are taking dot product of are representing different tokens and what are tokens?
6:36In simplest language, token is a word. In reality, tokens are sub words, but we can imagine that many tokens might represent certain words. So, let us say I have a semantic space like this, even if there are some dots here, these dots are basically end points of vectors and let us say these vectors are pointing into a certain direction of the space corresponding to animals. So, maybe this dot is about cat, this is a dog, this is a tiger what not, then there could be another set of vectors which are pointing into a slightly different region of space. So this we can call as semantic space which is definitely not two dimensional, it is multi dimensional, I am just showing a two dimensional space here and here these vectors may be corresponding
7:21to fruits like apple, orange, pineapple, etc. Now, clearly the cosine similarity or the cosine of the angle between these vectors, so what is cosine of 0? Cosine of 0 is 1 and if the angle is extremely tiny between the vectors, it means the cosine of the angle between those vectors will be close to 1 and that is called as the cosine similarity. So, if cosine similarity is high between these vectors what that means is the vectors might be talking about something that are close to each other or something that are related with each other. On the other hand if the cosine similarity is very poor if it is 0 or if it is negative then the vectors may be regarding completely different ideas so it makes sense to kind of consider the angle between these two vectors in
8:12deciding how much attention these vectors should pay to each other because ultimately the token which is coming from encoder and token which is coming from decoder are vectors that are residing in the same dimensional space. And if it is in the language of query and key, the query and key are residing in the same dimensional space and you are trying to take the dot product between them. So, a simple way to compare these vectors is take a dot product between those two vectors, the dot product value will be a high positive number if the angle between the vectors are small, if the angle between the vectors are 90 degree, the dot product will be a high positive number if the angle between the vectors are small. If the angle between
8:46the vectors are 90 degree, the dot product will be exactly 0. If the angle between the vectors are 180 degrees, the dot product will be a big negative number. And now if the vectors should pay attention to each other, you know high attention to each other because they are semantically related to each other, then naturally the dot product value will be a high positive number. So whenever we get a high positive number, we can directly kind of use that positive number itself as the attention score. So just simply take dot product as the attention score. So I have an interactive lab here. I will share this link with you, but essentially you can play with these vectors. So here we are doing the attention score calculation, which q dot k, q is the first vector, k
9:29is the second vector. In the language of large language models, this can be called as query and key, but we have not yet defined in this series what is query and what is key, which we will look at in the next lecture. But for all practical purposes, if these are the two vectors that are attending to each other, obviously you can see that the attention score is proportional to how close these vectors are in terms of angles. So as you rotate this, you will see attention score is highest when they are pointing in the same direction. So, just feel free to play with this, but this is very basic. But the problem with this attention score is the following, the naive dot product produces scores that can result in highly saturated softmax values when we convert attention scores
10:14into attention weights. So, first we have scores which are raw numbers right and then these scores are converted into probability distribution in the form of attention weights. And what are attention weights? Attention weights are basically resulting from softmax. So, there is one idea that, so here just look at this graph. On the y-axis we have attention score value, typical attention score size, on x-axis we have dimensionality. Dimensionality meaning when you are taking dot product between two vectors, what is the dimensionality of those vectors? In this diagram actually I have shown two dimensional vectors, but in reality vectors will not be living in two dimensional space, they will be living in hundreds of dimensions,
11:04a space that has hundreds of dimensions. So as the dimensionality of the space increases, when you go from 64 dimensional space to 128 dimensional space to 256 dimensional space, what happens is the attention score, which is resulting from the dot product of query and key, the value, the magnitude of that dot product of query and key, the value, the magnitude of that dot product keeps increasing. I'll show you exactly with a Python script how this happens, but basically, and this is assuming you are randomly initializing the two vectors, because when you are training the neural network, the vectors will be random numbers. So when you randomly initialize the Q and K vectors, and when you take the dot product,
11:44the resulting dot product value will be proportionally increasing with dimensionality. But is it linearly proportional? No. This number is proportional to square root of dimensionality. So if dimensionality increases from 64, like let us say dimensionality is initially 1 and let us say then later it became 64. So, dimensionality increased by 64 fold which means the magnitude of the resulting dot product will increase by square root of 64 which is 8. So, that is the reason why when the x-axis increased to 64 from 1 y-axis increased to 8. So, an order of magnitude, a factor of 8 increase. So now you can imagine, and one more thing is, these are raw attention scores, what happens to raw attention scores.
12:35But you might ask a question, why is this a problem? If attention scores increase in magnitude like this, why is that an issue? So let's open this Google Collab notebook. I just want to show you some things okay so first let me delete the current output and disconnect so this is a simple google collab notebook where i am defining one query and so this is one query so query is a random vector whose dimensionality is equal to the embedding space dimensionality so initially i am saying the embedding space dimensionality is one so one dimensional vector okay for simplicity i'll increase this keys however we have two keys so that's the reason why here it is np dot random dot rand of two comma dk two comma dk means
13:21whatever is the dimensionality of the embedding space, in that embedding space, we are creating two vectors. So can you imagine how many attention scores will be there in this case? Is it one or is it two? There should be two attention scores, right? The query.key, dot product between the query and the first key value and the dot product between query and the second key value. So you will have two two attention scores and when you convert these two attention scores into attention weights so the attention scores are nothing but dot product between query and transpose of key then attention weights are not that attention weights are basically attention scores passed through a softmax function. Now, before passing through softmax function,
14:07what if we scale the attention scores by dividing the scores using square root of d. So, here because the attention scores are increasing by square root of d proportional to square root of d, what if you divided the attention score, the raw attention score by square root of d, then ideally this should become like a straight line right so when it when i say straight line what it means is the magnitude of the dot product between q and k will not increase when the dimensionality increases okay that is what we are trying to do so let's first see the see this thing and then we will discuss the reason why this is a problem so the dimensionality is one you have one one query, you have two keys, which means you
14:47will have two attention scores, you are scaling the attention scores by dividing by square root of t, then you are taking the softmax. In fact, you will be calculating softmax of scaled attention score and unscaled attention score and you will print them. So when I run the code, this will print a few things. So, first query has shape 1 comma 1 key has shape 2 comma 1, 2 comma 1 because you have two keys. The raw attention scores between them are the dot product between those two random vectors. So, this is the attention score which resulted from the dot product between query and key number 1. This is the attention score which results from the dot product between query and key number two in fact you know we could even print the
15:29query and keys so print Q and we can also print K so you can see what the Q and K looks like so So, I will move this print statement about, so it can come before everything else is printed. So this is q, it is just one single number these are the K values random numbers okay so the raw score basically this point for two is the dot product between this and this simply multiply point nine zero seven multiplied by point four six because here dot product is simple multiplication because the dimensionality is one currently so now the rod raw scores are like this to attention scores when you can pass it through softmax you get the attention weights and you can see that attention weights are summing up to one
16:29the raw scores will not sum up to one 0.42 plus 0.24 is not one but attention corresponding attention weights will sum up to one 0.544 plus 0.455 now the scale scores in this case is exactly same as raw scores why Because square root of d is square root of 1. So, scaling does not do anything in this case, which means the attention weights calculated using scaled attention scores is not different from the normal attention weights without scaling. So, it is 0.544 comma 0.455. Now, let us increase the dimensionality, let us say my dimension space is 4 dimensional. Now, if my space is 4 dimensional you can see this is k sorry this is q, this is the first key and this is the second key. Now, when we take dot product, the dot product between query and the first key is 0.94, query
17:22and the second key is 0.94, query and the second key is 0.42. And when you convert the raw scores into softmax, this is the resulting softmax, look at this 0.62 comma 0.30. Now after scaling, when we scale the raw scores, we are dividing by square root of d, in this case the dimensionality is 4, which means square root of 4 is 2. So 0.94 will get divided by 2, which is is 2. So 0.94 will get divided by 2 which is 0.47, 0.42 will get will get divided by 2 which is 0.21 and when you take softmax of these numbers it will become like 0.56 comma 0.43. Still you might be wondering the scaling by dividing by square root of d did not do much what's the big deal nothing is happening but you will see what's happening immediately. So let's say now I don't want to print because when i increase the dimensionality the printed thing will be very big so let me
18:08remove the print part now i'll increase my dimensionality to 1000 and you look at what happens now when i increase the dimensionality to 1000 my raw scores are 252 comma 258 okay before scaling and look at what happens when i directly take the softmax of those unscaled scores so one value is 0.99 the other one is 0.002 basically this number this softmax number is completely dominating this this is becoming close to 0 and this is becoming close to 1. can you imagine why this is happening why the softmax core is core is getting saturated like this it's getting saturated because think about what is softmax softmax is e raised to something divided by sigma e raised to all the numbers right so e raised to 258
18:56divided by sigma e raised to two uh basically sum of e raised to 252 plus e raised to 258 e raised to 252 is negligible compared to e raised to 258 because e raised to 258. e raised to 252 is negligible compared to e raised to 258 because e raised to 258 is a very big number. e raised to 252 is very small number compared to e raised to 258. So what is happening is when the dimensionality is increasing these raw scores will keep on increasing. Why is that? When you are doing dot product you are taking dot product of lot more numbers. This time you are taking dot product of thousand pairs. Earlier you were doing dot product you have you are taking dot product of lot more numbers this time you are taking dot product of thousand pairs earlier you were taking dot product of only one pair when the
19:29dimensionality is one now you are taking dot product of thousand pairs which means there is a very likely very good chance that many of those many of those pairs will be you know positive numbers and those positive numbers will keep on adding up to produce a bigger raw score and if you directly take the softmax of raw score you get a saturated softmax scores on the other hand when you scale here when you scale you are dividing by square root of d which is square root of 1000 which is roughly 30 something right so you are dividing 252 by 30 something which will result in 7 point something 258 is also divided by 30 something, which will result in 7 point something. 258 is also divided by 30 something, which will be 8 point something. So now if you take the soft max
20:10of these numbers, the numbers are kind of better. It's not saturated numbers. Soft max is reflective of the original attention scores, because here the attention score says that kind of the weightage is kind of equal. But if you directly take soft max, it says the weightage is kind of equal but if you directly take softmax it says the weightage is completely unequal it says this guy is completely dominating than this guy but that's not true attention raw attention scores does not say that they say the weightage is kind of equal and here if you see this the softmax calculated using scaling after scaling you see the it's kind of equal 45 percentage versus 54 percentage here it is 0.2 percentage
20:45versus 99.7 percentage now this problem increases in magnitude when our let's say my dimensionality of the space is 4096 okay you will see the you will see this becoming crazier the raw scores are 1032 versus 1042 kind of close to each other right the raw scores are kind of close to each other but look at what happens when you don't scale if you don't scale the attention weights become weight becomes three times 10 to the power of minus five which is extremely close to zero and this is 9.999 basically if you talk in terms of percentage it's 99.996 versus 0.0003 something like that so uh so what the softmax is saying is these attention scores are such that this guy should be this is the dominant score and and the probability distribution is such that zero point
21:36zero zero zero zero one three versus ninety nine point nine nine nine nine nine six that is not good on the other hand when you scale this time you take the square root of 4096 and then you scale the scale numbers are like this 14.13 and 14 sorry 16.13 and 16.29 and look at the attention weights they are 0.46 and 0.53 basically 46 percentage and 53.9 percentage much more sensible attention weights right so this is the reason why we are doing scaling now think about it what happens if my attention weights are like this what is happening to my attention weights when you don't scale when you don't scale the attention weights are becoming it's it's kind of more and more resembling one hot encodings so the way it works is let us say my ground truth is this.
22:36Ground truth is equal to 1 0 0 ok and let us say because we had saturated attention rates my prediction became something like actually let me close a few tabs so that my writing is a bit smoother okay and let's say the prediction is something like this 0.99 comma 0.005 comma 0.005 do you think in this case the loss will be high or loss will be low so this is one example in which case you are let's say you are making the correct prediction this is prediction number one and let's say prediction number two is this 0.6 comma 0.2 comma 0.2 in all the in both of these cases the probabilities add up to one uh prediction is saying that the first the correct answer is let's say the token number one okay but in which case do you think the loss will be higher the loss will be obviously higher in
23:50prediction number two because prediction number one is like a highly confident correct prediction whereas prediction number two is not so confident correct prediction which means the loss will be higher when it corresponds to prediction number two and as a result gradient will be higher and if gradients are higher the network will get modified. Now imagine the process of training our neural network or training our large language model when we are training let's say when our model makes a prediction in the beginning for then for the large language model weights to be modified we need the predictions to be not so confident prediction in the beginning. But here look at what is happening. The raw scores are not so confident scores, 754, 728, 741, 739.
24:36Those scores are almost similar to each other. So it's a very under confident raw score. But look at what is happening because now we are taking softmax on these raw scores. This low confident prediction, the softmax is converting it into a high confidence prediction. See, the first one which is dominant, 754 is a dominant one, that is becoming 99.999 and remaining ones are close to zero. And as a result, what will happen is, although in reality the prediction that is made is not very confident the resulting loss will become so low because the softmax will convert this low confidence prediction into a high confidence prediction so that is the issue if you don't scale on the other hand if you do
25:14the scaling look at after scaling 31 percentage 19 percentage 24 percentage 24 percentage so it's back into a less confident prediction so this is the primary reason why we have to do scaling by dividing by square root of d i'll just show you visually once so that you are clear about this once more so here you can see four tokens as the potential next tokens the down cat sat etc so in fact here when we talk about softmax course we can talk from two different perspectives one is softmax probability scores corresponding to next token prediction or softmax course resulting from attention weights at an attention scores converted into attention rates okay so here we are primarily talking from the attention
26:00weights perspective but the same argument applies when we are talking about softmax course corresponding to what should be picked as the next token okay so in this case these are the raw numbers two four one three okay and dimensionality is four so two four one three when you uh do not divide by square root of d look at the uh softmax distribution 0.09 0.64 0.03 0.24 so the dominant one which is cat is becoming more dominant and as you increase the dimensionality you will see that cat will become more and more and more dominant you see so after certain dimensionality this cat is completely getting getting saturated because here cat was the dominant one in fact we can actually modify these numbers a little bit so let's say the cat let's say they are all
26:55close to each other but only slight difference 2.5 and down is also 2 so here in this case SAT is dominating a little bit so the is 16 cat is 16 SAT is 20 and down is changed down also to 16 in this case SAT is dominating a little bit but you can and that domination is not a problem if dimensionality was low if dimensionality was low SAT will dominate but not by exceeding exceedingly or not by exceeding degree. But as the dimensionality of the embedding space increases, the saturation of softmax becomes so high that the SAT, this SAT term when you convert softmax cores into softmax weights, it will become so dominant and this is not good for training our neural network or training our large language model.
27:42Now if you divide by square root of D, will see that softmax score is again back to or the attention weights are back to some a bit more sensible distribution so this is why we divide by square root of d so um the raw scores when you do not scale it takes an approach called as winner takes all, meaning whichever is dominant, whichever is winner, completely starts dominating. But when you scale it, this problem can be fixed. Okay, so this I have already discussed, so I am not going to spend a lot of time on this slide. Now, just to take a recap of what we have done so far, In fact, this lecture is almost ending. In the last lecture and this lecture, we discussed combined three attention mechanisms. First is additive
28:31attention, additive attention is aka Badano attention, V dot product with tan hyperbolic of W 1 s plus W 2 h. So, here W 1 s you can consider as the one of them is the query, the other one is the key and here it is additive because these two numbers are getting added. In dot product attention you if it was dot product attention instead of doing this w1 s plus w2 h you will simply do s dot product with h. In scale dot product attention you will do the same thing s dot product with h divided by square root of t although here we are just using the language of query and key which is a more generalized language, but essentially you take the dot product between those two vectors and take the square root of D. So, just to recap where we are at in this course we have
29:18only looked at this part so far which is the different attention score calculation mechanisms. We looked at additive mechanism, dot product mechanism and scale dot product mechanism. And as you already might know, scale dot product mechanism is by far the most popular and most frequently used. So in almost all the attention mechanisms that we will discuss from here on, you will see that what it uses is additive dot product. Sorry, my bad, scale dot product. And we are almost done with spending a lot of time on bucket number one from the next lecture onwards we will move on to bucket number two and further buckets as we move as we progress with the rest of the lectures in the series. So now we are in a good shape to discuss about self-attention mechanism from the next lecture
30:01onwards. So the takeaway from this lecture is very clear. In the next lecture, we will build self-attention mechanism by hand from scratch. And from today's lecture, I will share the slides link with you. I will also share this attention lab link and I will also share the Google lab notebook link. So please go through them on your time. So I'll see you in the next lecture. Take care until then.
← Part 3Additive Attention (Bahdanau Attention) | The Attention Family, Lecture 3Part 5 →Simplified self attention | The attention family series - Lecture 5