Why does Tanishq say reference-free RAG metrics can be misleading in production?

Tanishq says generic reference-free metrics do not scale well to high-risk settings and can give false hopes or false success rates, because a real agent serves one particular use case. His advice is to tune the judge prompt for each metric in libraries such as Ragas or DeepEval, swap the default OpenAI provider if needed, and decide on providers from your own empirical results.

• Tanishq says OpenAI judges have worked well for him and suggests trying Claude if budget allows.

▸In their words

“Don't trust everything which is written in the paper or in someone's blog, always try these things yourself.” Tanishq · at 22:46 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-18). Read the whole episode summary →

▸More from this episode