What are evals for AI agents, and how are they different from unit tests?

Evals are the QA step for AI agents, according to Tanishq in Gaurav Sen's mock interview. Where classic software had unit tests and a separate QA team, evals test the parts of an agent: whether the prompt leaks, whether personal data escapes during a tool call, and whether tools are called the right number of times. Unlike unit tests, there is often no ground truth, so teams build a synthetic or expert-made golden dataset to test against.

• Tanishq says the engineer who builds the agent should also run its evals, or brief the eval owner thoroughly.
• LLM-as-a-judge is one technique; quantitative evaluation is possible when enough numeric data exists.

▸In their words

“Evals is basically the QA for AI agents” Tanishq · at 3:04 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-12). Read the whole episode summary →

▸More from this episode