The attention family · Vizuara · Transcript & summary by WhipScribe
It gives you one seven-slot framework for placing any attention variant, so the next model paper reads as a change to a known part rather than new jargon.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“So you can see that the bucket number one and two and seven are exactly the same across these three models.” Instructor · at 17:47 — open in the transcript →
“If you are implementing attention with n squared complexity, the scale of computation that is required will be something which no machine can handle.” Instructor · at 23:50 — open in the transcript →
Vizuara's series The Attention Family starts from a problem anyone reading recent model papers will recognise: there are easily more than 50 well-known attention variants, from scaled dot-product and masked attention to grouped query, multi-head latent, flash, paged, ring and Kimi Delta attention. Dr. Sreedath Panat, a Vizuara co-founder with an MIT PhD and an IIT Madras degree, argues that memorising the list teaches nothing. The opening lecture teaches no single mechanism; it hands the learner a map for sorting all of them.
The map has seven buckets: how the attention score is computed, which tokens may attend to which, how heads and the KV cache are arranged, whether the cost stays quadratic in sequence length or drops below it, how the kernel runs on a GPU, how position is injected, and which domain the mechanism serves, from text to images, video and graphs. Any new paper, in this framing, improves one of those slots, and a genuinely new idea in even one of them counts as a new attention mechanism. Applied to Llama 3, DeepSeek V3 and Kimi Linear, the scheme shows all three sharing scaled dot-product scoring, causal masking and the text domain, while differing on heads, position encoding and sub-quadratic tricks.
The rest of the series moves from foundations (Bahdanau attention and Attention Is All You Need) through position encoding, the KV cache, grouped query and multi-head latent attention, then sparse and linear sub-quadratic designs, and finally GPU kernels, Swin Transformer windows and graph attention. Learners who already know the softmax(QK^T/sqrt(d))V formula can skip the early modules, though Panat recommends watching in order. Colab notebooks, slides and interactive websites are promised along the way.
Vizuara's Dr. Sreedath Panat sorts every attention variant into seven buckets: scoring (how the raw attention score is computed), connectivity (which tokens may attend to which), heads and KV cache, sub-quadratic cost, GPU implementation, position information, and domain (text, image, video, audio or graph). Two mechanisms can match in some buckets and differ in others, and the series maps every concept it covers back onto these seven.
In the opening lecture of The Attention Family, the Vizuara instructor says there are easily more than 50 reasonably famous attention types introduced by different papers, and shows a slide of roughly 35 to 37 of them, including flash, paged, sliding window, cross, bidirectional, linear, Bahdanau, deformable, ring, grouped query, multi-head latent and Kimi Delta attention. He expects new papers to keep adding more.
Vizuara's lecture maps the three models onto its seven buckets. All three use scaled dot-product scoring, causal (masked) attention and the text domain. Llama 3 adds grouped query attention, rotary position embedding (RoPE) and flash attention. DeepSeek V3 uses multi-head latent attention with a decoupled RoPE and also flash attention. Kimi Linear shares multi-head latent attention with DeepSeek V3 but adds a 3:1 hybrid with Kimi Delta attention and uses no position embedding (NoPE).
The Vizuara instructor explains that with n tokens, every token acts as a query and every token as a key, and a dot product is taken between each query and each key, giving n times n, or n squared, operations. Sub-quadratic mechanisms such as sparse attention bring the exponent below 2, which he calls essential for long inputs like video with many frames.
Vizuara's attention family series puts flash attention in its GPU-implementation bucket, not in scoring or connectivity: it is about how attention is run on GPU hardware so inference is much faster, alongside paged attention and ring attention. Llama 3 and DeepSeek V3 both use it. The series covers these kernels in its final modules.
The Vizuara series runs in modules of rising difficulty: foundations first (Bahdanau attention and the Attention Is All You Need paper), then position encoding, the KV cache, paged, grouped query and multi-head latent attention, then sub-quadratic designs such as sparse and Kimi Delta attention, and finally GPU kernels like flash and ring attention, Swin Transformer windowed attention and graph attention for images, video and graphs.
No. The Vizuara instructor expects two groups: learners who know neural networks, backpropagation, multilayer perceptrons, embeddings and softmax but not transformers, who should start from the next lecture, and those already fluent in the softmax(QK^T/sqrt(d))V formula, who can skip early lectures by title. He still recommends watching in order because self-taught learners often have gaps they have not noticed.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:01Hi, welcome to a new lecture series from Vijwara. This lecture series is about the attention family. I have named it as attention family simply because there are many different types of attentions that are existing that has been proposed by different papers and different models in the literature. You may have heard of scaled dot product attention. You may have heard of masked attention. There are many other attention types, group query attention, ring attention, paged attention. We have flash attention and many more. And literally if you try to count the number of different types of attentions, you will see that there are easily more than 50 plus reasonably famous types of attentions you will see that there are easily more than 50 plus reasonably famous
0:46types of attentions that have been introduced by different papers. So hence I have put the name the attention family to capture all of these under one umbrella or under one roof. So this lecture series is to give you a very good idea about not just the attention that was introduced in the attention is all you need paper but also to introduce you to how people have come up with different types of attention mechanisms that are more efficient faster etc and i also want to not just lay down upon many different attention mechanisms in front of you because that will not probably teach you how to think of your own attention mechanism if you want to design something of your own. From first principles, how can you think about how to build an attention mechanism that may improve
1:37a certain very specific aspect, right? So this lecture series is about giving you first of all an idea of all the different types of attention mechanism, trying to bucket them in very specific categories, teaching you what those buckets are, and trying to implement some of these attention mechanisms by hand on code, etc. And at the end of it, you should feel comfortable to probably think of some aspects that you could improve yourself, right? So my name is Dr. Sridhar Panat. I am one of the three co-founders of Vishwara AI Labs. I'm a PhD from MIT and I graduate from IIT Madras. So this lecture series is purely dedicated to all of you, whether you are a student or industry professional, if you want to master attention mechanism completely end to end.
2:26I have been wanting to conduct a series like this for a very long time and now I'm glad that I have managed to put this together. I'm not exactly sure the total number of lectures that will be there in this series but I suspect it will be easily more than 10 or 12 lectures but maybe much more than that also because I am not yet sure whether I'll be combining you know discussing about a few different attention mechanisms into one single lecture or whether I'll split it into multiple lectures that I have not decided yet but I'll show you the different attention mechanisms that we will be discussing from a deep theoretical perspective and also from a practical perspective in this lecture. So in this slide I'm just
3:08showing you simply three models LAMA3, DeepSeq V3 and Kimi linear model and here you can see three terms GQA rope, causal, MLA rope flash attention, KDA 3-2-1 hybrid and NOPE. So GQA stands for group query attention, ROPE is rotary position embedding, causal attention, MLA stands for multi-head latent attention, KDA is Kimi delta attention, NOP is no position embedding. So basically if you look at any model from literature or any popular model from industries, you will see that certain components might be different. Like you might find two different models that may use causal attention and group query attention, but instead of rotary position embedding, some model might use sinusoidal position embedding.
3:58Or here you can see some models might use multi-head latent attention. Some other models may not use multi-head latent attention. So first of all, what are these things? What is group query attention? What is multi-head latent attention? Why is it used? What is flash attention? Is flash attention fundamentally changing something within that attention architecture itself or is it coming at the time of implementation when you try to implement this for inference in GPU? What is paged attention? What is Kimi Delta attention? There are so many questions, right? Simply because at the moment, maybe for many of you, these are just terms. Or maybe you are familiar with some of these architectures. I mean, some of these attention types,
4:35but not every single one of them. So I just want to show you one slide which may make you feel a little bit uncomfortable, but I'm just going to lay down a bunch of names, okay? These are simply jargons which may intimidate feel a little bit uncomfortable but i'm just going to lay down a bunch of names okay these are simply jargons which may intimidate you a little bit but please don't worry about it i'm just showing you what all things we might cover overall as part of this series but in a very structured way so flash attention paged attention sliding window attention sinusoidal embedding causal masking cross attention bi-directional attention linear attention badano attention which is very
5:09old the attention that is used in Swin transformer the window then deformable rotary position embedding Delta net group query attention then lot of them Kimmy Delta attention scale dot product uh then multi-head attention itself and i'm just showing you around i think 35 or 37 different names here ring attention self attention multi-head latent attention a lot of names okay i'm sure like you have maybe heard of half of them maybe many of the names themselves are very new and when a new paper comes out tomorrow i'm sure they will introduce something new in terms of attention mechanism and i'm sure it's going to improve some aspect of either inference or maybe it is going to improve the ability to have long range attention in a better way maybe it will reduce the compute but increase the
6:06needed memory so there are many many different concepts so firstly the question that we have to ask ourselves is first of all is it worth learning all of this second if we if we are really interested in understanding all the different attention mechanisms how exactly can we can we bucket them because clearly these are not standalone mechanisms right there are something all the different attention mechanisms, how exactly can we bucket them? Because clearly these are not standalone mechanisms, right? There are something similar between these mechanisms and there are maybe boxes in which we can fit these attention mechanisms. So maybe a better way to learn about these mechanisms is to understand what are those boxes. So in today's
6:43lecture, today's is purely an introductory lecture. I'm not going to teach you any specific attention mechanism in today's lecture. However, my goal is to lay out what you can expect in the upcoming lectures in this series as far as different attention mechanisms are concerned. So I want you to have a very, very good look at this image where I'm showing you seven buckets in which we will be discussing about different attention mechanisms. Sometimes when you compare two different attention mechanisms there may be difference or similarity across more than one bucket okay and I'll show you what these buckets are and even if you don't fully understand some of these terms it's completely okay today because I'll be anyway
7:31starting from the next lecture from the very basics so don't worry about understanding every single thing every single jargon that I'm mentioning in today's lecture first is about scoring this is the first bucket so scoring means when we calculate or compute the attention score how is it computed there are many different ways in which we can compute the attention score so the first bucket in which we will be classifying the different attention mechanisms will be based on scoring second is connectivity meaning which tokens are allowed to look at which other tokens so and there are many different attention mechanisms based on this connectivity between tokens third is the attention heads themselves how are the heads arranged
8:13is it multi-head attention or is it something else then if we have kv cash how big is the cash itself and if you don't know kv cash again don't worry we'll be discussing about that but this is the third bucket this is about the attention heads themselves fourth is we you might have heard that the complexity of computation of attention mechanism scales with the square of number of tokens because the attention weights matrix which we will anyway discuss in the upcoming lecture even if you are not familiar with that it scales with the total number of uh with the square of total number of tokens so that's the reason why here we say quadratic so you may often hear the term n squared n raised to 2
8:55n squared because if you have n number of tokens uh each token will become query all of those tokens will also become a key the query and key has to you know have to take dot product between queries and keys and the total number of such dot products will be equal to n multiplied by n which is equal to n square so it is quadratic complexity however there are attention mechanisms which have managed to bring down the complexity to sub quadratic so it won't be n raised to 2 it will be n raised to something less than 2. So there are some attention mechanisms we can bucket into the category of sub quadratic. Then based on implementation, actual implementation in terms of GPU implementation, there are differences, there are different attention
9:40mechanisms. For example, flash attention is a very good example how you implement it in practice so that is the fifth bucket sixth bucket is about the position information like how attention knows in what order tokens are existing there are many attention mechanisms where we don't embed position information many other attention mechanisms have naive learnable position embedding many of them have sinusoidal position embedding. Many of them have rotary position embedding. So there are many different mechanisms through which we inject position information into the attention mechanism itself. And finally, is the attention mechanism belonging to text domain, video domain, image domain, or let's say audio domain?
10:22To which domain are we adapting this attention mechanism obviously if you are thinking about attention mechanism between text and images there are differences because image definitely has a spatial structure right a today if you are looking at an x y um black and white image it definitely has a 2d structure which means there has to be some difference in the way attention mechanism is implemented for an image versus a text a sequence of text tokens so that is the seventh bucket so now when we categorize attention mechanism maybe some of them will fall in multiple of these buckets so we may have attention mechanism that is falling uh that is common um that has buckets number one two and three common between them but maybe bucket number four four, five and six, four, five, six and seven are different for that.
11:07When you compare those two attention mechanisms, maybe there are attention mechanisms where out of these seven buckets, six of them are similar. Six categories are exactly the same, but the seventh one is completely different. So that is how we should ideally be able to visualize attention mechanisms. More details you will understand as we go through the lectures and today also i will show you a couple of architectures where the bucket themselves differ a little bit let's move on to the next slide so if you really think about what the attention mechanism is trying to improve, when I say attention mechanism, the attention mechanisms proposed in new and new papers, when a new paper comes out tomorrow, it will try to improve one of these things.
11:56It will try to improve what is being computed, meaning how exactly are we computing the attention score between tokens. That will definitely be one aspect that will be improved and that will belong to the bucket number one and two so if you go back to bucket number one and two first one is about scoring second is about connectivities so first scoring means the raw attention score connectivity means which tokens are allowed to attend to which other tokens we'll this we will visualize this and discuss this in detail in upcoming lectures. So don't worry about not being able to fully visualize. I'm just laying out the structure of the course. Then if you think about cost, there is compute cost, there is memory cost.
12:37So there are certain attention mechanisms which will tremendously improve, let say the compute cost kv cache is a great example and bucket wise it will belong to bucket number three and four then some of the attention mechanisms will simply try to improve the implementation for example when we are performing inference using gpu that inference could be much faster so flash attention is a great example you may have heard of paged attention also then we have the order and the input so order and the input means order is are we injecting position information or are we injecting the order in which tokens are appearing in the sequence are we injecting that information into the attention and input is what is the input
13:25type are we having image text or graph there is also graph attention so are it could be that also or is it video right so if you really can come up with an improvement in even one of these aspects apart from what people have already done then we can say that you have come up with a completely new attention mechanism so my hope is that towards the end of this lecture series all of us will be comfortable thinking about new attention mechanisms I'm not saying that it's a low-hanging fruit obviously not because many people are working in this field and it's already there are many attention mechanisms that people have tried but yet at least we can have a better mental map of how this whole field is
14:11looking like so if you take a very specific example of llama 3 okay you can see that it is making some very specific choices let me just open the drawing so in terms of scoring attention score it is using scaled dot product attention which is very standard i'm not saying this is something extremely special but there is naive dot product there is just uh additive scoring is there so it could have taken any of these two also but obviously as we all know scale dot product is the probably the best scoring method then lama3 has causal attention which means the future tokens uh the current query can only see uh keys from up until that point it cannot look into the keys from the future so it's using causal masking basically masked self
14:57attention then we have attention head so lama3 is using grouped query attention then it uses rotary position embedding rope right so that is a position information but it could also be sinusoidal alibi or it could be a simple naive learnable position embedding it could be anything then finally in the implementation part it's using flash attention so now if you look at these one two three five aspects okay llama 3 is following aspect like from scoring it's following this aspect from mask part it's following this aspect causal if you look at the attention heads part it's following group query attention position embedding wise rotary position embedding and kernel implementation wise flash attention and that is how the llama structure is
15:45however there could be some other attention mechanism where instead of rotary position embedding maybe it's implementing sinusoidal position embedding and obviously that is not the only difference between architectures okay then total number of trainable parameters could be different the number of parallel attention heads could be different the number of attention blocks in series could be different the number of attention blocks in series could be different so there could be many differences between architectures but at a fundamental level these items themselves could be different between two different architectures all right now if you were to compare let's say these three models I i'm just showing you the seven aspects which i
16:25discussed so if you compare llama 3 versus deep seek v3 versus kimmy linear model and if we compare these three across the seven buckets which i mentioned so seven buckets are scoring how attention score is calculated connectivity which tokens are allowed to attend which which other tokens heads and kv cache sub whether it it is sub-quadratic or not, how the actual GPU implementation looks like, how position information is injected and in what domain, whether it is text or image. You can see that there are some similarities between LAMA3, DeepSeq v3 and KimiLinear. For example, the domain is text domain, so the bucket number 7 is common. Then LAMA 3 and deep seek v3 there it is using rope rotary position embedding but there is a
17:09difference so this deep seek v3 use deep decoupled rope rotary position embedding which we will discuss in the later part of the lecture series similarly if you look at implementation you can see some similarity both Lama 3 and deep seek v3 also use flash attention then if you look at DeepSeqv3 versus KimiLinear you can see that they both use multi-head latent attention and if you look at the connectivity meaning which tokens are allowed to attend which it all of them use causal attention or masked attention which is very common and obviously scoring wise meaning the attention score themselves when you calculate all of them are using scaled dot product attention. So you can see that the bucket number one and two and seven are exactly the same across these three models.
17:54But the rest of the buckets are attention mechanisms or difference in architectures in this way, you will have a much easier time kind of understanding about different models. So if tomorrow a new model comes out, you will not feel intimidated about the fact that, oh, it's doing something different. That won't be an issue because you will be able to categorize it into one of these buckets. That feature will be part of one of these buckets. So now before we start you might have one question okay if you are a complete beginner who has no idea what an attention mechanism even is you have not read the attention is all in it paper or you are not familiar with this field at all. Is this something which you can follow?
18:47So I just want to address that question. So I assume that there will be two sets of people who will be following this lecture series. First set of people will be people who are very familiar with deep learning, basically people who are familiar with neural networks, who know what is, basically who are familiar with everything except transformer architecture. You may or may not know transformer architecture, but you are familiar with everything except transformer architecture. You may or may not know transformer architecture, but you are kind of familiar with neural network architecture, back propagation, what is a multi-layer perceptron, what is a vector embedding, what is softmax operation, so some of these basics, right.
19:19So if you are at this level or a level before this, I would highly recommend you to start from the very first video very first video meaning next video onwards because there i will be starting with the very fundamentals or foundations of attention mechanism however there is also a good chance that you might belong to a bucket where you are already familiar with the transformer architecture you you don't need a lecture on what is attention mechanism you don't probably need a lecture on kv cache maybe you don't need an attention on what is scale dot product attention so when you look at the formula context vector as a function of query key and value is equal to softmax of query dot key transpose divided by square root of d multiplied by value that equation if you are
20:01very familiar with that equation a few lectures in this series you will be able to comfortably skip okay so um and that you will easily understand without even going through the lecture if you just look at the title of the lecture you will understand whether you are familiar with that topic or not if you are completely familiar feel free to skip that topic and move ahead to later lectures otherwise i recommend you to start from the very first lecture itself and what all things are we going to cover roughly? I'm not saying that exactly this will be the order in which I'll be covering, but more or less this will be the order. So I plan to cover the foundations in the first one or two lectures where we will build the attention mechanism itself from scratch. Then
20:42I want to have some discussions about position information how attention mechanism learns about position information i'm 100 sure a good fraction of people watching this video are already familiar with this so if you want to want to have a refresher go through those lectures which i will be releasing uh after this but if you don't want that refresher completely feel free to skip it then I want to have a lecture on kvcache which is very important I will also discuss about sparse attentions I will discuss about linear models like recurrent neural network I'll discuss about attention from the context of implementation on GPU which is where we will discuss about flash attention then towards the last part of the series I will also discuss about beyond text beyond text meaning
21:29images videos graphs how the multi modality comes into the picture when it comes to attention mechanism and every single time whenever we are discussing about any concept we will try to map it into the seven buckets which I have mentioned okay so this is going to be the structure of the lecture series. And the first module, in the first module, I'll be covering the attention mechanism itself, starting from Badano attention, how people thought about attention mechanism even before the attention is all you need paper and how things completely changed when attention is all you need paper was introduced. And we will also discuss whether someone else had introduced something even before that paper so this will
22:09lay out the foundations for those of you who are who are kind of feeling intimidated by the fact that your your foundations regarding attention mechanism itself is not very strong so if that is the case these lectures will cover it for you then in the lectures two and three as i mentioned i will be i don't want to call it the lectures two and three, as I mentioned, I will be, I don't want to call it as lectures two and three. I would rather call it as modules two and three. The reason is I may take more than one lecture per module. So I would rather keep the nomenclature as module two and three as of now. So I will cover about position, how position information is injected, how KV cache is implemented the practical
22:46concerns with respect to kv cache memory consumption versus how it improves compute etc we will also discuss during these lectures paged attention multi multi head latent attention group query attention etc and then and of course during these discussions we will also refer to the corresponding models which introduce these mechanisms for the first time and other popular models are that are famously implementing them even till date then we will discuss in the modules four and five how people figured out to reduce the complexity from n squared to something less than n raised to 2. So the full attention map is basically n squared of n squared complexity because as I mentioned n queries and n keys you have to take dot product between each and every single one of them.
23:37So n times n, n squared that is the complexity but there are attention mechanisms like sparse attention which reduces the complexity which is extremely important when it comes to implementing attention in, let's say, videos, because videos have a lot of frames. If you are implementing attention with n squared complexity, the scale of computation that is required will be something which no machine can handle. So we will discuss some architectures which try to reduce the complexity we will discuss about sparse attention we will discuss about uh kidney delta attention etc and a few more different architectures then in the last three modules we will discuss about on gpu implementation meaning uh flash attention we will discuss paged attention, ring attention. We will discuss about attention mechanism that was implemented in papers like Swin Transformer.
24:30It was very, very interesting, actually, the way they managed to reduce the complexity without too much hurting the long range dependency between tokens. Then we will also discuss about graph attention. So these things will be covered in the last three modules. So I have set the modules in a very specific order so that the complexity is increasing from initial lectures and it goes on increasing towards the later part of the lecture. So if you follow the lectures in the sequential order, you will find it very easy. And I'm also planning to construct interactive websites using which you can, you know, you can interact and you can plug and play numbers and you can understand whatever concept we are covering in an even better way.
25:09So I'll be sharing those links also with you. So as I mentioned, there are two ways to go through this lecture series. One way is skip what you already know, only follow what you don't know. But the risk there is that maybe you may have knowledge gaps which you have not identified so even if you feel like you have understood something thoroughly maybe there are knowledge gaps so revising will not hurt so what i personally recommend is go through the videos um sequentially so as i mentioned each uh like the total number of videos it's a little bit variable but i'm hoping that each video it each video actually won't be half an hour it might be longer than that so although i have initially written here roughly 25 30 like
25:50i wanted to keep it under 30 minutes now that i think of it each video can be over 30 minutes also so please don't hold this against me but i recommend you to go through every single video in the sequential order and you will benefit from it a lot and these are the kind of mechanisms that we will cover and this one two three four five six seven is not the lecture series sequence this is just the bucketing from the scoring bucket from the connectivity bucket from the headset kv bucket sub quadratic bucket and the remaining three buckets there are different attention mechanisms which primarily fall into or different concepts which primarily fall into each of these buckets so we will discuss these also
26:31along the way and this is the only image that I want you to you know store in your head or just take a screenshot and keep it keep it handy whenever you hear about a new attention mechanism at the end of this lecture series, my hope is that you will be able to immediately connect with it and say, oh, this architecture is different from this other architecture in terms of bucket number two and bucket number six. So if you are able to visualize in that way, I'll be very happy and I'll feel very successful in this attempt at teaching the attention mechanism family to you. So in the next video, we'll be covering the very basics of attention mechanism how it evolved including badano attention. I have covered it in another
27:13lecture series previously last year however I still feel like in the context of this lecture series it's completely worth it going through that. So in the next video we'll be covering the very basics of how attention mechanism even came into existence and every single lecture probably won't have a coding part but in many lectures there will be coding parts so i'll be also sharing the google collab notebooks with you so i may or may not code everything from scratch because if i code from scratch things will prolong and maybe the lectures will increase its duration to one and a half hours etc so i'll decide it on the go in some lectures i made code certain modules from scratch in some lectures i will have code already written i will just give you a proper walkthrough and you
27:55can also go through the code sequentially block by block once i share the code with you and i will also be sharing of course the lecture slides with you so that you can go through them on your own time. So that's all I wanted to cover in today's introductory lecture. I am really looking forward to seeing you follow very rigorously and truly understand attention mechanism in and out. And in future, I hope some of you will be able to come up with some better aspects when it comes to attention mechanism implementation or the even the basic attention score calculation or some aspect I hope you will be able to propose something new or even write a new paper submit it into some workshop or conference and let's see so that's my hope and
28:41all the best and I'm looking forward to seeing you in the future lectures thank you so much bye