Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A short, whiteboard-style answer to a common AI interview question: what an LLM's context contains and where memory fits in.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“So whatever I am considering inside the model to make a decision that is considered as a context.” Tanishq · at 2:26 — open in the transcript →
“So memory is the stored thing.” Tanishq · at 5:55 — open in the transcript →
This eight-minute mock interview on Gaurav Sen's channel puts Tanishq, a 2025 graduate with some internship experience, at a whiteboard to explain context in large language models. Tanishq uses a trip to Japan as the example: an agent starts with the system prompt, the tool descriptions and the user's query, and each tool call, first for hotels and then for flights, adds its output to what the model sees on the next step.
Gaurav then asks what happens when context runs to millions of tokens. Tanishq offers two answers aimed at keeping the agent from getting confused and hallucinating: keep separate summaries in a memory layer, one per traveller, or split the work across several agents so each holds only its own share, which Gaurav likens to microservices. The last stretch separates context from memory. Context is what the model holds while answering one query, and memory is stored data, kept in a database such as Redis or Postgres, that gets pulled into context when a question needs it, like a past doctor's visit or last trip's preferences.
The candidate in this mock AI interview, credited as @aloo_explains, a 2025 graduate with some AI internship experience who is looking for a full-time role.
In Gaurav Sen's mock AI interview #2, candidate Tanishq defines context as everything the model considers when making its next decision. For an agent, that starts as the system prompt, the tool descriptions and the user's query, and grows with each tool response. In a Japan trip example, the hotel tool's output and then the flight tool's output are both added before the model picks its next step.
Tanishq gives two options in Gaurav Sen's AI interview series. One is to summarise and store the results in a separate memory layer, for example five summaries when planning trips for five people, rather than holding everything in the model's context window. The other is a multi-agent pattern where planning and summarisation agents each hold only part of the context. Both aim to stop the model getting confused and hallucinating.
Tanishq tells Gaurav Sen that context is what goes into the LLM to answer one particular query, a kind of runtime memory, while memory is data stored outside the model. When a query needs it, such as a healthcare bot asked about medicines prescribed at the last visit, the relevant memory is extracted and loaded into the context.
Partly, in Tanishq's answer on Gaurav Sen's channel. When Gaurav asks why memory is not simply called a tool call, Tanishq agrees that retrieving it is a tool call, but says memory refers to the storage itself. That storage needs a database, and Tanishq names Redis and Postgres as common choices.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Hi sir, good morning. Tanishq, hello, yes, just call me Gaurav. When did you graduate? I graduated in 2025. And do you have any experience in AI? Little bit of internship experience. I'm looking for full-time option, sir. Okay, okay. So I want to know what would you do if your model had a massive amount of context. Can you actually explain this on the whiteboard? Yes, yes, sir. So, Tanishq, could you tell me what context means with respect to LLMs? So, for example, if we have a user query like, you know, plan a trip to Japan, this goes to our system, right? The agentic system which we have built, which basically contains the LLM, it contains the system prompt, and it contains the system prompt and it contains the tools now if we talk about
0:45context here what is inside the context as of now right as of now what we have is the system prompt and it also contains the tools descriptions and all of that plus the user query these are the things which are currently in the context of the LLM and it plans the action it takes the action of you know calling a certain tool let's say for first tool is hotel now this goes to our system the system basically again contains all the information which we had plus the information of the hotels right so all this information I am calling this let's say I1 so now this system has I1 plus hotel information. Now after this context what it is going to do we also need some flight planning and all of that so it
1:30decides that I need to call a flight planning tool. So now this basically systems makes this call LLM decides that I want to call the flight planning thing. Now it calls the flight planning and we again the same action is performing. Here we are performing a certain action and we have an output of you know information from the flights. Basically this is the information that we got. Now if we talk about what is in the context now. So from the pattern we see that previously we had user query, we had system prompt, we had the tool descriptions. After taking one action we have the action which we perform the output of that particular action right we have it in the context now after this when again LM decides to make a certain
2:14call tool call or take certain actions now we have that information as well in the context so now the information is basically I1 plus hotel plus flight. So whatever I am considering inside the model to make a decision that is considered as a context. Basically the user query, the system prompt, the tools and the tool responses. So all these things are basically part of a context that I have to manage. So this is basically it. Do you have any follow-up question or any clarification here? What if the context gets very, very large? Like what if I have millions of tokens in the context? So let's say if you have millions of tokens, right? Let's say, for example, I have to plan a trip for, you know,
2:58five different places or people, five different people. Then I will have a lot of tool calls. I will have a lot of context which I have to store. Then are two options either i am going to you know make a summarization kind of system which is going to basically summarize based on five people so i will have five different summaries and in this case we cannot just handle it with you know the context window of a model we need to have a separate memory layer added to our agent tech system so currently as i have explained in the previous answer that we only had you know system prompt and tools and all of that we didn't had any memory layer explicitly as per the memory we only had system prompt but right now if i have to you know call this multiple times and for
3:39multiple people then i will have to you know store summaries for all the five people that I have. Why am I doing this is to stop the agent from hallucinating because as you said that if I have you know millions of context then all of this context is getting stored inside the context window of the model and it is getting confused it might get confused and we have an hallucination problem. So one way is this I create summaries five different summaries store it in five different you know places and then call it basically for every person at every single time. Other thing is we move to a multi-agent pattern. I can, you know, talk about it in detail if you want or because multi-agents will have different agents for particular thing.
4:18So for planning, we'll have different agents for, you know, then summarization will have different agents. So context will gets divided. Basically, it is a separation of concern basically wow sounds like microservices yeah it is the same yeah all right close yes okay okay so this is what the context window is yeah but what is memory what are we storing in memory is it like a database or what is it yeah that's a really good question so if we talk about in the context of a single agent system basically in general the context window is about 1 million if we are not crossing this then everything gets stored inside the model's own weights itself basically inside the context window but if we talk about you know the difference between context and memory
4:58is it's very simple so context is basically what is going inside the llm what i am giving it as a context to answer to a particular query this particular query so that is basically the context what is memory memory is basically if i have you know stored something here for example let's say i have made built a chatbot where i have a chatbox and there are previous chats basically how it is happening in chat gpt or cloud so whatever is happening inside this single thread in a single chat session is basically going inside the context of the memory along with the system prompt and other things which we have but if i want to you know if i look into an example of a health care and if i ask the bot can you pull up the last visit what
5:40were the medicines prescribed by the doctor in that that case, I need to basically do what? I need to basically extract relevant memory to fill in the context of the model to answer to my query. What are we extracting basically from? The memory, which we have stored. So memory is the stored thing. Context is the runtime memory. You can call it a runtime memory, which model have while answering a particular query. So would it be correct for me to say that if it's a trip app then my last trip context will also be taken? I mean is the actions of my last trip or the things my preferences in the last trip are they being stored in memory? Yeah, if it requires. If your current query requires you to extract something, if you say in the query like you
6:22know when LLM is asking you a question in the previous example we saw when we were defining agents that llm can ask you a question to give it more context so if it asks you a question that you know can you tell me your preferences and you reply to it like you know can you see my preferences from the last trip which i followed i am looking for the same then it is going to extract and that what it is basically extracting from is the memory and it is loading it extract and that what it is basically extracting from is the memory and it is loading it into the context. So why is this not just a tool call? Why do you call it memory? It's a tool call for a database right? I mean, yeah, it's a tool call itself but whatever so the memory part is what we are
6:58storing. So it is a storage basically. So for that storage purposes, we need a database. Some people you know use Redis, some people use Postgres. So these are some examples which people regularly use for storing the thing. So this is the basic difference between context and memory. Awesome. I think selection is for sure. How much do you want in salary? 500k? No, salary. Partnership. This was right. Everything is good son. Let us say I have a model with more than 100 billion in terms of weights and I also have a small language model. Which one do you think I should use depending on the use case? So sir, you cannot you know for you because you are a small you are small. You can... Okay, okay.
7:50So can you tell me what is a large language model? So sir, any model which is large.