How do you set up LLM judges for the three RAGAS metrics?

Gaurav Sen describes three separate judges, one each for faithfulness, answer relevance and context relevance, rather than one prompt that scores everything out of 100. Each judge gets a system prompt naming its single check plus the relevant inputs: context and answer for faithfulness, user prompt and context for context relevance, prompt and answer for answer relevance. If all three return yes, the eval passes for that query.

• Test cases can then be run against the three judges as a regular evaluation suite.
• Gaurav Sen notes that in practice all three inputs are often passed to each judge together.

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-18). Read the whole episode summary →

▸More from this episode