Research Paper Walkthrough: RAGAS Evals

0:00Yeah. Tanish, would you like me to just read the abstract because I think that will add a lot of context and then we can also summarize and move ahead. I'll just read this abstract for one minute. Everyone you can read this with me. We introduce RAGAS or RAGAS which is Retrieval Augmented Generation Assessment. So a framework for reference-free evaluation of rag pipelines, rag systems are composed of a retrieval and an LM based generation module. Okay. So basically you retrieve something from some sort of database and you have some sort of generation being done by the LM. Okay. In between, there is something called augmentation, which is basically taking your retrieved context adding it to the original user query and sending it for generation so that's what they're saying and provide elements with knowledge from a reference textual database okay most of the

0:57times the sql that you fire gives you text does not give you images so okay which enables them to act as a natural language layer between a user and textual databases reducing the risk of hallucination so this is good things about rag rag is grounded rag reduces the chance of hallucination because the lm does not need to imagine an answer it can just read what you have sent in the context and give an answer so now what's the problem okay what's the current situation every paper almost starts with now what's the problem? Okay. What's the current situation? Every paper almost starts with this. What is the problem? Evaluating RAGA architectures is however challenging because there are several dimensions to consider.

1:35The ability of the retrieval system to identify relevant and focused context passages. So if you're retrieving something from a vector database or a graph DB, whatever you're retrieving something from a vector database or a graph DB, whatever you're doing, the challenge is that are you retrieving the right things or no in your rag pipeline? What if you retrieve the wrong things? What if you retrieve irrelevant information? Then it becomes bad context. And the ability of the LM to exploit such passages in a faithful way. So you got the context. Let's say you got the right context. Is the LM actually able to even use the context to create the required response? So that's part two of the problem.

2:17One problem is getting the context. Second problem is can you use the context or the quality of generation itself? So you got the context. You can use the context, but did you generate the answer in the right format? Did you generate the answer as the user had expected you to generate? Is it satisfying all the guardrails? So these are the three main problems with RAG that they are mentioning here in the abstract. With RAGAs, we put forward a suite of metrics which can be used to evaluate different dimensions so they are specifying what evaluation metrics should be made without having to rely on ground truth human annotations this is the part which tanish mentioned which is really really important that's why they've put it in italics right

3:04without having to rely on ground truth human annotations means you don't need a bunch of data analysts you don't need a bunch of human experts to go through your rag pipeline okay they are going to tell you some metrics these metrics can be evaluated without people sitting and doing this themselves manually we posit that such a framework can crucially contribute to faster evaluation cycles. They are guessing that this is going to happen, which is especially important given the fast adoption of LLMs. Now, when did they come up with this? 2025 April. I think the paper is even older than that. Yeah, this is the updated version. The paper is 23 or 24. 23 or 24. So in 23 or 24, they said that LMs are moving fast.

3:46We think that they will move faster. And they are right. They got 2000 citations because of their correct guess. And this is a useful framework. So what is our intent today? What is our agenda today? Our agenda is to go over this paper and understand in a deep level, how do you evaluate a rag pipeline? Okay. You can't just take this and directly apply it to your rag pipeline. Maybe you can, but this is going to give you the right thinking framework required for you to go tomorrow or whenever you want to go to office and make a rag pipeline yourself. So that's why we have picked up this paper. By the end of this session, you can expect that you will have a good idea of how to evaluate any rag pipeline.

4:31Okay. Okay. So I think the abstract makes it clear why, what they do, why we want to know it. If you want to have a look at the introduction, usually we follow this practice. The introduction is describing the abstract in a much deeper way. It describes what is the problem. So, Tanishq, what do you think? Should I go over this and would that help? No, I think we can go towards the related work. That would be better. I think whatever you have said is already being covered so great great so this is explaining that you know the stuff that is out there right now is not sufficient we have found something special current situation is so and so so we will skip that because we are now in 2026 and 2024 the paper has come out so we have moved way ahead of this we are convinced that

5:26evaluations are important okay so uh related work estimating faithfulness using lms this is basically lm as a judge okay um and automated evaluation of text generation systems again this is there's a bunch of papers which can help you understand where the research was in 2023 okay so basically yeah so people were basically trying to use chat gpt or other models to basically score the outputs based on 0 to 100 or five star rating sort of thing and they were very dependent on reference based scoring so and the problem with reference based coding is that for any use case that you might be working on, do you know all the queries that people are going to ask, you cannot cover all the bases, right?

6:16So you can only cover the basis that you can think of. So you can only make a data set of from those samples. So that is one problem which they were addressing with reference free scoring so yeah we will we are going to discuss that so this is this is super useful and important to know like why would you why would you do this um okay okay so this was just again giving more context on what has happened until 2024. Okay now evaluation strategies. This is where the actual paper starts. We consider a standard RAG setting where given a question queue a user has asked where is my order or can you suggest an itinerary or you know what is the best mobile to buy under this budget. Okay you got that question Q. The system first retrieves some context,

7:08C of Q. What is C of Q? It's basically retrieval. The function when given your retrieval function, when given the query Q, what context is it retrieving? That is C of Q. Okay. So this might be some PDF files. this might be some text, whatever be the case. I mean, it's not going to get PDF files. It's going to give you text or some proper parsed data. Basically, yeah. Context, yeah. And then it uses this context to generate an answer, A S of Q. So A S of Q actually depends on C of Q also. So when building a rag system, we usually do not have access to human annotated data sets or reference answers. What does this mean? It means that when you're building a rag pipeline in your office, you don't have a data set with human annotations that this is,

8:00this data means so-and-so, or you don't have reference answers. For example, if someone says, give me the best mobile to buy between 20,000 and 30,000 rupees. Do you have the answer for that? It's possible that your customer support has never received such a query. It's a new query. They might have received a query for 30,000 and 50,000, for 25,000 and 35,000, but you don't have a reference answer. And even identifying a reference answer is difficult. So we therefore focus on metrics that are fully self-contained and reference free. So like Tanishq mentioned, you don't need reference answers. You don't need examples for these metrics to work. Okay. And for the people who might, you know, leave in between before we talk about all these

8:50things that these things usually, you know, when you're actually working on a production grid application, you need to tweak all these things a lot. Usually reference free things doesn't scale that much or you can't use them in a high risk setting so keeping in mind uh yeah just note it down so yeah then tanishq is very aware of our attention spans these days so yes that's a that's a good point uh tanishq okay okay any questions until now folks? Is the motivation for going over the paper clear? Okay, do think about what are the things which Ragas will actually have and then you can post some questions. Okay, I'm seeing some interesting ones. Is there any some, I used LM as a judge in a project where I tried to make an enforcement

9:45layer. So we'll take this a little later. It's not very related, right? So, uh, folks only ask relevant questions for now. Later, we will answer in comments, any general questions you have. Okay. So first, what is the first metric that they're proposing faithfulness in every rag pipeline faithfulness is important what is faithfulness faithfulness is refers to the idea that the answer should be grounded in the given context what does this mean you got a query you now need to generate a response before you generate a response you take context okay you pull that from some vector database or some graph or some SQL database. You got the context, you got an answer. Now you need to see how faithful the answer was to the retrieved context.

10:34So if Gaurav got four pages in refund queries, and the refund query clearly says that in India, refunds will be processed for the first seven days. Did the answer take that into consideration when it was a refund query if it didn't then faithfulness is low if it did then faithfulness is high yes and and when you're like working on a production grade application or in a company you need to give citations along with whatever answer that your llm is giving and what context did it use so this faithfulness matrix is faithfulness metric like is like there it's relevant to that context that whatever you have taken as a context to give the answer you have to give citations as well it helps there it helps people getting at least some amount of interpretability of how the system is actually working.

11:27So this is one very helpful metric. Most of the times when you have any answer which you're quoting, like if you make a claim, like Tanishq mentioned, you just say that this is the citation. Like even in research papers, there is that brackets one, square brackets one like that. You'll see even in most of the cloud or open ai their answers for technical answers they show references at the bottom so this is what is actually happening ref answer relevance refers to the idea that the generated answer should address the actual question which was asked so for example i asked tanishq tanishq uh can you tell me what is the best mobile to buy between the cost of 10 000 rupees to 20 000 rupees i want it delivered in

12:13seven days what a laptop is much better to buy and you can buy some laptop and say 70 000 80 000 yes yes this is the thing like customer is king in matters of taste and also with their query. So LLMs also need to respect that, right? They have to be relevant to the answers have to be relevant to what the question actually was. So because LLMs have a way of drifting away to avoid that one evaluation metric clearly is answer relevance. Was the original question Q related to the answer A-S of Q? Now comes the third one and this is interesting, most of you must have guessed it by now. Context relevance. What does it say? Refers to the idea that the retrieved context should be focused. Containing as little irrelevant information as possible.

13:10What does this mean? It means that if I got the query by a mobile. If I retrieve all the context. If I take the whole database and I put it in my augmented query. Then of course I have retrieved all context. is wrong in that well the precision is very low okay means you have taken irrelevant context what happens if I take little bit of context but I miss out on other important context means that precision is very high but it is so precise that you are actually missing out on relevant context so recall is bad I can't remember what the correct context is. So knowing this, there is a complete metric for this kind of a thing. Recall precision. Instead of that, forget all that. Rack specific metric, context relevance.

13:58And like, yeah. And they're mentioned in the paper, like they have made this point very clear that there are only three things that you need to focus on the query, the answer and the context. Right. And based on these three things, all these. Faithfulness. So just remember that context relevance is all about query and the context, which is retrieved. Faithfulness is about answer and the context context which is retrieved faithfulness is about answer and the context which is retrieved so keeping in mind these two things will help you a lot in segregating or designing the prompt for the judge what is really good about this is these basic concepts i think most of us understand even even non-engineers will easily understand this and they can go to their teams

14:47and say that, look, dude, we have created a rag pipeline. Can we talk about its answer relevancy? Can we talk about its faithfulness and can we talk about context relevance? Because if these three things are high, then at least it passes the smell test. If these three things are low, then every rag pipeline requires all three of these to be high. If it's not there, then you need to work on the pipeline. You'll figure out where is the root cause, why is context relevance low, why is answer relevance low, why is faithfulness low. So we explain how these three quality aspects can be measured in a fully automated way and which is available through the OpenAI API. Okay, so Tanishq is defining this very, very well.

15:35The faithfulness and the answer relevance in just a few words. Okay, it's very concise information. Tanishq just mentioned that if you are thinking about faithfulness, just think about, did the answer, did the answer actually use the context? Okay. So technically what you'll say is to estimate faithfulness, we first use the LM to extract us set of statements. Okay. This is a, you got the answer. You got the context. This is the context. The aim of this step is to decompose longer sentences into shorter and more focused assertions. We use the following prompt. Okay, so they're describing the method. Given a question and answer, create one or more statements from each sentence. Okay, question and answer.

16:19Okay, I think this part we can kind of go over or summarize because it is going to get quite technical as to how they did it. Tanish, do you want to summarize how a person can calculate faithfulness? And should they calculate it or should it be like yes or no? Yeah, so usually when they defined it, it i hope i'm audible there is some issue with my network yeah i'm good yeah so i when they defined it they defined it using a formula sort of thing that you know the state supported statements divided by the extracted statements but usually what works better is you making a binary decision uh This was back in 23, 24. So, but currently when you are designing an LLM as a judge and whatever metric that you're designing, it is better to design it in a sense that it is giving

17:13a binary response yes or no zero or one. So when you're designing faithfulness, just think about giving the judge, these two things, the answer, whatever answer your pipeline gave and the context which you retrieved through the vector DB or the similarity search, ANN search, whatever you are using. So think about these two things and all these three metrics care about these two things only. It's all about, you know, permutation combination of these three things, answer, query and context. So based on these three things, you can formulate, you can think about more metrics as well. In this paper, they have only talked about three, but if you see their current repository, they have made a lot of changes. They have added some more

17:53metrics because it's all about these three things, what you're retrieving, what you're asking and what you're answering. It's that simple. And you can, yeah, you can like, it's like human nature. Yeah. Yeah. I think think like if you just look at the components of rag folks you'll see where is the data actually being uh pulled from that's where you can check context yeah who's generating that's where you can check uh the faithfulness and whether you're doing it the right way that that's where you can check for answer relevance. So pretty simple. Tanishq has said that you don't need to go over this formula. You're in luck. You don't need to do an MTech on math and AI to go through this. This was a proposal from their side

18:45and it's a great idea, but it didn't really in practice, you know, much better is just binary. Yes or no, why? Because LLMs have subjective answers. It is very hard to quantitatively say that this is very high faithfulness or very low answer relevancy. It is either yes or no microsoft also does the same thing emoji happy or sad don't tell us you're kind of happy like there are two emojis in microsoft when you're giving feedback it's either or it is there is no there is no right uh but you'll see in Linux or other systems where there are 5 emojis. Knowing this, I think the basic idea is quite clear. They have also clearly explained how they did this with the wikival dataset. So we can actually run this.

19:40If you have stuff like this at home i'll just show you what you need for this your stuff like this at home seven lakhs only cheap as hell no it's actually nice for organizations to maybe fine-tune your own model yeah gigabyte has solutions for that you can run experiments also if you're a researcher um again i don't know why I'm promoting this so much, but I'm just genuinely telling you that Gigabyte can help you run experiments. This is the AI top atom. If you're buying this, thinking of buying this yourself, then you better be super rich because seven lakhs. But if it's an organization, they can afford this. That's it. That's how simple the paper is. Look at the size of this paper. It's just five pages.

20:27Because the idea is simple and it works. It's got like 2100 citations and it's used across the industry. When there's a rag pipeline, people talk about what should I evaluate. This is it. For now, it's it. Like Tanishq mentioned, there are a few other things also new evals have come up coherence completeness but if you understand these three then you will understand those also reasonably well yeah new evals and also it's more about you know how you are designing these judges as well yeah so uh these the the point which i said in the middle that, you know, don't always go ahead with, you know, trusting these reference-free metrics. Because when you are actually building a eval set or eval suite for your company, you need to give them the context of the use case that you are building it for.

21:19Because the agent that you have built, it is not generic, right? It is for a particular use case. So when you are doing evaluation as well, you have to stick to a particular use case and these generic metrics or generic reference free kind of approaches they don't usually work well because they might give you you know false hopes false success rates so in this paper also they have defined this false success rate that you know it will give you false positives and all of that. So always think about when you go through these the repo of these libraries like Raga's or deep eval or any other thing, you'll see you'll see that you can tune the prompt, you can change the prompt which is being used for

22:14a particular metric. So you can always tune that you can always change that for your particular use case. And one more problem which you might come across is that they are all of these libraries are you know, by default using open AI care using any other provider you have to change that. But I think I have tried open AI a lot. The judges are not it works well. Maybe if you have you know more money and spend more than go ahead and try clod as well it's all about empirical results don't trust everything which is written in the paper or in someone's blog always try these things yourself and make a informed decision don't be one more thing that we should probably touch upon is um faithfulness answer relevancy and context relevance.

23:07Quite a few people ask, you know, what should I pass? So firstly, folks, these are going to be computed by large language models, right? So what you need is you need a system prompt for that judge large language model. So one is creating the answer. Another is judging the answer. The judge needs a system prompt saying that you are checking for faithfulness. You don't say check for faithfulness and answer relevancy and context relevance and give me a score out of 100. No, no binary single responsibility. One. So you'll have three judges, one for faithfulness, one for answer relevancy, one for context relevance. What do you pass to these judges? Of course, there's system prompt. Plus you say for faithfulness,

23:48this was the context that I retrieved. This is the answer I got. Is it faithful? Is the answer faithful to the context? In context relevance, you say this was the prompt I got from the user. This is the context I retrieved. Is the context relevant to the original query? And in answer relevancy, you say, is the answer actually relevant to what the user asked? So you pass the prompt and the answer, but you can also pass context in this. Usually all three are passed together. Okay. And what do you ask the LM in the system property? You say, give me a binary yes or no. Is it faithful? Or just give me a JSON response of response equal to response colon yes or no. Or Boolean true or false. I'll read this and I will.

24:38If all three say yes, eval passed. If all three say no, then for this query, the eval has failed. And then you can create test cases uh and run them against these lms judge yeah and always go through the paper just yeah remember like this one thing that always go through the paper and if i give you one rule of thumb which you know which you can cut it out and make a reel out of it is that when your answer is wrong when you think that your answer seems wrong you should always check for faithfulness because the context might be wrong if your answer seems irrelevant like the user is asking about something and you are responding about something else then check answer relevancy if answer is missing context

25:25then you should go towards checking context relevancy so always think about all these things in in a pair that if you are checking context relevancy you care about the question and the context if you care about groundedness or faithfulness you care about answer and the context if you are checking answer relevancy you care about question and the answer so do think about it we also have some you know notes which i used for uh some other video which i was making so if you would like, we can score between 0-100. Okay. Amazing. Thank you everyone for coming. It's a short paper. I hope the information though was worth more than a few hours. If you would have tried to read this yourself, it would have taken you time.

26:20We were able to compress this, condense this into a 20 to 30 minute discussion. The other thing is if you're looking for more in-depth AI knowledge, of course, we have a live cohort that we take at AI Eng, interview readies for online courses. Tanishq has a cohort coming up in September. September, I can tell you the exact date. I can actually share my screen here to give more context. Let's share this. Okay, yes. So this is the Dragon Agents cohort. We have taken one cohort previously. Tanishq has taken a cohort previously. It had very high user feedback ratings. And we have, you know, customer testimonials here if you want to go through them. It's an eight-week cohort. And the main idea is that you build hands-on experience, hands-on skills, so that you can transition to AI roles.

27:31Forward deployed engineer, solution architect, AI engineer. This is the kind of role that you can move into. What we are covering here, what Tanishq is going to be going over primarily is first week is basics. You might be from, you may have a lot of experience around AI or you may have zero experience around AI, but there are some things which are always good to know about. For example, what's a context window? What's 300 billion parameters? What is 200 billion parameters? What is a hundred million parameters, what is 200 billion parameters, what is 100 million parameters, what is attention, what is vectors, what's tokens. So just code terminology so that you can then converse with AI practitioners comfortably.

28:18Reading papers also then becomes possible because you know the terms, you know the meanings, so you're able to grasp things quickly. Then comes, okay, then becomes possible because you know the terms you know the meanings so you're able to grasp things quickly then comes okay the basic idea of a large language model pre-training and post-training and inference so what happens in these three steps followed by some coding so tanishq is really into hands-on practical you know visible uh stuff so tanishq prefers claude i think is it is it claude yeah yeah so tanishq will be using claude and you will also be using any lm that you like to see this happen in real time okay that's a different experience you know exactly what each term means when you see it happen in front of you. Followed by RAG, something that we covered at a

29:08high level today but this is going to be in a lot more detail. RAG has chunking strategies, it has embedding models, it has vector databases, it has ways in which you can actually pass the data so that is going to be not only taught in terms of theory which is Saturday but practiced on Sunday so that on Monday when you go back to office you can actually perform better yes that's the goal it happened last time it will happen this time too and again advanced RAG basically covers more advanced concepts in real time. How do you make it more accurate? How do you consider evaluations? So stuff which is important for production grade applications. OK. Again, of course, coding is there so that you can take this knowledge

30:01and implement it. Followed by week four, which is RAG architectures and specialized types. Here there's knowledge augmented generation. I think someone mentioned OKR or OK some, which is a very, it's a variation of RAG. Tanishq covers agentic RAG also, right? Graph RAG, all sorts of RAG, but more importantly like you once you know the fundamentals you'll see that catching up with this is very easy okay finally i think the most important bit if you're thinking about building applications agents single agent systems the difference between lms and agents tool calls react pattern and making type safe agents. Okay. This makes it production grade. And when you go to multi-agent systems, you are actually moving a little into the future, but we have future prepping you also.

30:57Yes. So orchestration, multi-agent systems, memory, most of the things that you will see or are seeing right now, which are bleeding tech are covered here. Okay. And you will be seeing in future as well. Yeah. My guess is 2027, but Tanish does not like making predictions, but yeah. Any prediction now? Are you going to make an exception today? No. No, okay. Yeah, but multi-agents I think would work by next year. I think if you make architecturally well, like if you focus more on the system design of AI, you can make it work. Right, right. So what I'll do is I'll also just, I think there are some comments coming, which are, um, where is the, um, you know, can you pin this? Definitely get it, getting it on personal message.

31:58So, okay. I have shared the link with you, uh, It's been context engineering, memory, and evaluation. Again, core topic for engineering. Quantitative evals, evals with LLM as a judge. What are the things that you have to consider when creating evals? What is context engineering? So absolutely core, followed by a capstone project. So once you have all of this with you, you can actually build a capstone project. So once you have all of this with you, you can actually build a capstone project. And I and Tanish recently, we were working on a pothole app, which got national media attention. And the reason for that is not because we are the world's best engineers. It's just that you can build things in a day or two,

32:41as compared to two years ago, when it would have taken you two months definitely three or four years ago it would have taken you three months four months right this is insane a better app and all of that everything included it would have taken you more than three four months three four months which means uh entire team of engineers would sit down and make it in one month and so that would have been a cost which would be like a startup cost now you can do it in literally one or two days okay and uh why is there national media attention right now because it changes people's lives so in the capstone project what we strongly encourage you to do is choose something which has an impact in your life either it's a company project so So you, you know, do that.

33:28The company is happy. You are happy, or it's a personal project. So an expense tracker or health tracker, something which you care about personally or a community project, which is like, you know, pothole tracker or something which helps people get their Aadhaar card or national ID that is good for the community. You build a community and you serve the community together so this is what is then yeah just to add one more thing that it's not only about you know using AI in a better sense it's also about what do we have in our mind to build basically it's not about just some generic statement that i can use cloud better than you guys it's not like that uh it's it's more about uh you know making informed decisions while designing the system as well we don't completely rely on cloud

34:18to you know design the app for us then write the code as well, then do everything. We make decisions, we design the system and we use cloud as a tool. Not as, yeah. No, absolutely. And Tanish, one more interesting thing is that a lot of people, I think, have this concern that, oh, this is very expensive. One of the things is you can try to reimburse this program from your company. You can click this link. We will help you connect with your manager, with the relevant details, with a brochure. So if you are working in an organization, this is great. The second thing is we feel like if you start this program, you're going to stick around and you're finally going to actually meet the goals that you had for 2026.

35:03So we have a seven day money back guarantee if you feel like the after the first class you feel like you know this is not working out for me go ahead we will have zero questions asked and we will be sending you back the money uh that you have paid for this program minus the payment gateway charges which is basically something like depends right if it is striped then it's like three%. If it's bank transferred 60 rupees. So we don't keep any of your money. It's striped, which keeps the money, but yeah, it's a seven day money back guarantee. There's zero risk basically. Okay. Um, and you get 20 live classes with an instructor who's there to clear your doubts. We also have TAs with us who will be clearing your doubts constantly at any point that you post.

35:48Maybe not at 1 a.m. in the night, but it'll be cleared pretty fast. 90 days of implementation support. So if you have any questions after that, also we are with you. You can ask the people from the first cohort. Any cohort, basically. Any cohort. Past three cohorts, you can ask anyone. Yes, we are here with you. It's a community only that is being built, of course, of serious engineers only. Okay, and certificate of completion is also there. You can have a look at the FAQs. Often people have questions like this. Lifetime access to recordings? Yes. Class timings mentioned here, 9am to 10 10 30 a.m ist who should join the program it is for people who are looking to build production grade ai applications okay so if you're

36:32in college no you don't need to right now if you are uh you know in marketing or if you are in sales you don't need to right now you're look we are looking at engineers mainly. You need to have some software engineering experience. If you're an engineering manager, if you're a product manager, you can consider it. Engineering manager, definitely you can join. But everyone else, you need some coding experience. And are there any prerequisites? You need to be able to write code in at least one language. Okay, everything else I think is asked a little more rarely. You can download the cohort brochure here. You can also see the email template here for emailing your office. And if you want to learn ML from scratch,

37:21this is not the cohort. We are focused on engineering, stuff that you can actually do in office. ML research is also great, but that would be a research-based program. This is called an AI engineering cohort, not a AI research cohort. So there are great programs out there. I think Vizsura has a great program for research. You can go and join that if you're looking to build a transformer yourself. Here, you're looking to build a transformer yourself. Here you're going to actually use your transformer at work. That's what we are going to focus on. Okay. Amazing, amazing. So we have, I think nearly 500 students now who have actually gone through this. This is outdated and I'm really looking forward

38:01to this cohort also by starting on the 12th of September. Tanishq, any thoughts before we walk into the sunset? Oh, yeah. One more thing. If you're looking to contribute to an app, right, and you're a college student or you're a working professional, whatever, it doesn't matter. We have open sourced the Pothole Detector, right? You can check this also out okay this is getting international media attention if you contribute to this you have a chance of getting attention from companies there's a lot of attention here it's just 17 stars guys what the hell is going on it's got like 14k likes on twitter a lot of attention but we need real engineers to actually make this happen we are getting connected by various government

38:50organizations you can change india if you focus on these kind of problems yeah we don't only focus on you know talking about the problems we focus on solutions yes we don't just focus on making money and making this a sustainable effort we actually focus on the mission also so yes thank you everyone for for tuning in and we'll see you soon tanishk any any parting thoughts that you want to share nope i think you have covered everything it's just that i am like if you are a student and you know you want to have mentorship and all i have opened up some few few slots uh but that is for serious students who have three months of time to spend in learning and all if you have you can we'll attach a link

39:39in the description so yeah amazing amazing all right thank you tanish thank you have a great day ahead we'll see you folks. Thank you for coming. Thank you for tuning in. There's 40 of you here. It's like a full classroom. I always feel proud to have an audience, which is excited about these things.