Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A short, whiteboard-style answer to the interview question of how you would prove an LLM app is reliable enough for healthcare.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“reliability equals to observability plus evals” Tanishq · at 1:12 — open in the transcript →
“Evals is basically the QA for AI agents” Tanishq · at 3:04 — open in the transcript →
This eleven-minute mock interview on Gaurav Sen's channel puts a candidate named Tanishq in front of a hospital that wants to use AI to cut costs without ever giving patients wrong information. Tanishq's answer rests on one formula: a reliable AI system needs both observability and evals, checked before it reaches production.
Observability, in Tanishq's account, is the familiar logging habit applied to agents. Instead of plain logs, a team records agent traces: changes to the system prompt, every tool call and its response, and the decision the model made from it. Those traces are what evals run on. Evals play the role a QA team once played for ordinary software, testing for prompt leakage, leaked personal data and tools called over and over. Tanishq picks three metrics to start with: answer relevancy, groundedness, which measures hallucination against the supplied context, and tool sequence accuracy, shown with a flight-booking agent that must fetch user data before searching flights.
The practical side covers who does the work and what it needs. The engineer who builds the agent should also evaluate it, against a synthetic dataset or a golden dataset made with a subject-matter expert, using techniques such as LLM-as-a-judge. Groundedness needs the retrieved context, such as RAG chunks or tool responses, and tool metrics are better measured with precision and recall than with plain accuracy.
The candidate in this mock AI engineering interview, asked how a healthcare company could use AI while keeping its answers reliable.
In Gaurav Sen's mock interview, candidate Tanishq says AI is non-deterministic, so reliability has to come from two practices: observability and evals. Observability records what the agent does, and evals test those records before the system goes to production. Tanishq argues that doing both is what makes an AI app usable in high-risk fields such as healthcare or finance.
Tanishq describes agent traces in Gaurav Sen's AI interview series as the agent-era version of logs. A trace stores changes to the system prompt, which tools were called and how many times for a query, each tool's response, and the decision the model made from that response. The point of storing them is to evaluate the agent later, since evals cannot run without this record.
Evals are the QA step for AI agents, according to Tanishq in Gaurav Sen's mock interview. Where classic software had unit tests and a separate QA team, evals test the parts of an agent: whether the prompt leaks, whether personal data escapes during a tool call, and whether tools are called the right number of times. Unlike unit tests, there is often no ground truth, so teams build a synthetic or expert-made golden dataset to test against.
Tanishq names three: answer relevancy, which checks that the response addresses the user's actual question; groundedness, which checks that the answer sticks to the context it was given and does not hallucinate; and tool sequence accuracy, which checks that tools run in the expected order. A flight-booking agent that searches flights before fetching user data would fail the third check.
Tanishq's recipe in Gaurav Sen's AI interview #4 uses the user query, the bot's response, the system prompt and one extra input: the context the bot used to answer. In a RAG system that context is the retrieved chunks, such as the top five. For an agent it is the tool responses, plus any reasoning the model produced. High groundedness means the agent is following that context properly.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Hi sir, good morning sir. Yes, good morning. So, Tanishq, do you know what we do? Have you read about our company? No sir, actually I... No, it's okay. I expected that. Thank you. Our company works on healthcare. So, we have patients who use our app, look at their disease diagnosis, connect with doctors. And we as a hospital are very particular about giving only reliable information to our patients. So, you understand reliability and not making mistakes is something that we have to do, otherwise we'll get sued. So please tell us how we can use AI to cut costs, automate processes, but still keep our patients. So if you want reliable, right? If you want useful, basically, and you want it to not give wrong answer,
0:42then, sir, there is this thing called sir evals i have started learning it recently can you describe that in more detail can you just give me a brief demo on the board so tanish how do i make an app which is very reliable very secure and still uses ai so as you all know that ai is non-deterministic but there are still two things that you can do to ensure reliability in your AI agent or AI system so in every agentic AI or AI application reliability equals to observability plus evals if we can ensure these two things before going into production or you know at the time of testing and all of that then we can ensure that our system is reliable and it can be used at high risk areas like health care or finance whatever we are talking
1:29about okay now could you define what do you mean observability is it similar to what has been happening since forever yeah yeah it's the same things but here what we are observing is different so we talked about an agent right so an agent has a lot of nuts and bolts. It has a system prompt. It has tools, right? So whenever I'm making a decision, so I have the instructions stored here and I have all the tools listed here, right? So I need to store the response of the tool, how many times the tool is being called for a particular query and all of that. So I am storing all these things that comes under the umbrella of observability previously we used to store logs and all of that right here we call it agent traces so we are
2:13storing agent races and agent races consist of you know changes in your system prompt tools whatever tools you are calling whatever response you are getting whatever decision LLM is making based on the tool response and why are we taking these traces? What we debug them later or do we like make metrics out of them? What happens? Yes. So why are we storing it, right? What is the point of storing all this is to evaluate them, right? So if we are not storing like what is the decision making that LLM is doing? What are the tools being called? Is there any issue with tool calling and all of that? If you are not storing that, then we won't be able to do evals right we won't be able to evaluate our system on that okay now what is evals you the
2:51observability is helping power evals so what is that so previously you know in a classical software dev setting we used to have unit test and all of that right QA there were people in QA who used to do testing and all of that evals is basically the QA for AI agents right so whatever we are storing here in QA who used to do testing and all of that. Evals is basically the QA for AI agents, right? So whatever we are storing here, so QA used to happen on apps. They used to test different settings, different things, right? So here we use evals to test different nuts and bolts of our AI agent. For example, if prompt has any leakage or something like that, if during tool call there is any PII information getting leaked,
3:25or if the right amount of tool call is happening, if certain tool is not getting called again and again, despite giving the same information back. So there are multiple things we need to evaluate for our AI agent. So that is basically what evaluation is. So if I have to go to my engineers and tell them observability, I think they can go and implement it. But what about evals? Is it like unit tests? Is it something else? So evals, basically, I can share you, you know, let me just make a simple diagram. Here is your LLM, right? It is basically connected to a certain set of tools, correct? It has a system prompt, right? I'm taking a very simple example here. So we have an LLM, it's connected to certain tools, and it has a system prompt. I'm taking a very simple example here. So we have an LM, it's connected to
4:08certain tools and it has a system prompt. Now, what is your topmost requirement when you're making an application? Is that the output which is coming out of your agent is correct. You want it to be reliable, you want it to be relevant to whatever the user asked. Second thing is whatever your agent is answering on whatever query it is answering it should you know use the particular context which you are offering through the tools this also ensures reliability the third thing is that whatever tools your agent is calling it is not calling them again and again despite giving the correct answers or despite giving the same response these are the three things which you need to get upon now what are those three things called right in terms of you know when we do evals and all of that what
4:54are those three things called the first one is answer relevancy here you ensure that whatever answer is coming whatever your agent responding to a particular user query, it is basically responding to that particular query itself. The meaning is same. If I am asking you about what is an apple, you are not telling me basically what is a banana, right? You are telling me what is an apple basically. So it calculates the answer relevancy. The second thing is groundedness. Now groundedness is basically the second problem that I talked about that whatever your agent is answering for for a particular user query it is catering to the context which was given to it and it is not you know making up answer it is not hallucinating. So here you
5:36calculate the hallucination basically you can consider it as hallucination calculation. The calculation well the third thing is tool sequence accuracy so let's take that for example whatever you know whatever we are discussing here regarding the flight booking agent and all of that so when you are trying to book a flight what are the particular tools that you will have you will have you know one tool can be get user data yes one tool can we do a web search for flights and the third tool can be a payment gateway where you are making the payment for the flight, right? What if your agent directly goes ahead, does a web search? Yeah. Like if your agent directly is directly doing a web search without getting the user data, then it's a problem, right?
6:16So that is what we are handling at tool sequence accuracy. So you collect some data on, you know, you define some data that this is the sequence of the tools which should be there. And then you test your agent if it is following the same sequence or not. So this is the third thing. There are more metrics which you can do, you know, completeness, correctness, and all of that. But if you cover these three bases, and if an agent is working well on these three bases, then you can consider it as a reliable application. And who's going to do this evals? Is it humans or is it a machine? Like, is it automated? Yeah. So at the time when we were building software dev applications, QA team used to be separate. They used to test an application and all of that. Here, whoever is
6:56trying to build the application is the one who should test the application as well. So that is a change here. So eval should be done by the engineer who is building the application as well or there you know needs to be a proper kt session for the eval person to know what is happening inside the system what is being observed because this is the important part to go through evals this is the first step you know following up on evals so this is the thing and if we talk about you know how do we do this right what system do we use? So these are things where we discussed at the time of QA and all. We used to have some unit tests written, and we used to test our application based on those tests only.
7:33What is happening here is we don't have a ground truth. At most of the times when we are trying to build an application, it is not necessary that we have a ground truth. So we make a synthetic dataset on whatever we think is right, following a particular pattern, we generate a synthetic truth so we make a synthetic data set on whatever we think is right you know following a particular pattern we generate a synthetic data set or we create a golden data set which is by a subject matter expert and this is the most important part that you need to sit down with a subject matter expert to create a data set and you need to evaluate your agent on that particular data set this is how you know testing is happening these days there can be llm as a
8:04judge as a as a you know as a technique if we have the correct amount of data numeric data then we do quantitative evaluation as well okay but how exactly do i calculate this answer relevancy groundedness what are they yeah that's a really good question so what do we need to actually do to calculate these so for answer relevancy as i have told you that this is basically on you know if my bot is answering to that particular query only so I will be using the exact user query and the response and if you want to make it you know more useful you if you want to add more dimensions then you can also use the system prompt basically so if you have these three things then you can calculate answer relevance because it is based upon you know know, whatever your bot is responding to, what question is it responding and what are the instructions given to the bot.
8:50So this calculates the answer. OK, and what about groundedness? So for groundedness, what all do you need? So you need a user query. You need the response from the bot, what it is responding. You need the system prompt. And one more thing, which is extra in terms of, you you know when you are trying to test groundedness that extra thing is the context which the bot is getting to answer to the query so here we need context as an extra thing and this context can be anything this context can be the tool response which you are getting this context can be the thinking which the bot is doing so whatever context it is following to answer to this user query is needed to calculate grounded so in in RAG, this context is going to be the chunks that we pick up?
9:27Yes. In RAG, whatever top five chunks that you're getting is going to be the context and you have the user query system prompt and everything else that you have. Agents, the tool responses can be the context? Yes, the tool responses because that is basically how the agent is getting the context as well. So whatever I'm, when I'm booking a flight and all of that. So the context which I'm getting is from the tool calls itself. So we consider tool call response as a context when we are calculating groundedness for agents. If my groundedness is high, what does it say? If your groundedness is high, then it says that your agent is actually following the context properly. It is grounded in the context.
10:02It is grounded in the tool call responses. And what do we need for this tool sequence accuracy is basically whatever we have collected here in the observability, we use the same thing. So we use the tool calls which have been made by the assistant. So there are user and there are assistant, there are two different things basically. Assistant is basically what AI is outputting. So whenever my LLM says that you need to call this particular tool and our system makes that tool call, so we get the whole, we need the tool calls. And here I have loosely written accuracy, but actually you need to use recall precision and all of those metrics. Accuracy is just a loose term to represent all of those things. I think I'll make my system reliable now.
10:40Will this help me calculate how many hours are there in strawberry? No. That's the thing with LLMs, not with your system. That's the problem of the brain, not the machine. But tool calls can't be done for count the number of strawberries. Then why are you building a healthcare app? So how can we integrate AI into our hospital such that it is highly reliable, secure, safe, good for our patients, and still there is some automation? Sir, hire more nurses, sir.