podcasts / Gaurav Sen / episode / question
Question

Should an LLM-as-a-judge give a yes-or-no verdict or a score for RAG evals?

Answered from Research Paper Walkthrough: RAGAS Evals · Gaurav Sen · 2026-08-18

Tanishq says that although the RAGAS paper defined faithfulness as a ratio of supported to extracted statements, a binary yes-or-no verdict works better for LLM judges today. Gaurav Sen adds that LLM outputs are subjective, so it is hard to say quantitatively that faithfulness is very high or relevance very low, and he compares it to Microsoft's two-emoji feedback prompt.

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-18). Read the whole episode summary →

▸More from this episode

Q1What is the RAGAS framework for evaluating RAG pipelines?Q2What is faithfulness in RAGAS evals?Q3What is the difference between answer relevance and context relevance in RAGAS?Q5How do you set up LLM judges for the three RAGAS metrics?Q6Why does Tanishq say reference-free RAG metrics can be misleading in production?Q7Which RAGAS metric should you check when a RAG answer looks wrong?
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →