Is a re-ranker worth the extra latency in RAG?

Gaurav's answer in Gaurav Sen's mock interview is usually yes. Re-ranking 20 to 25 chunks adds about 50 to 100 milliseconds, which he considers worth it for a roughly 5 to 10 percent accuracy bump measured on metrics such as faithfulness and answer relevancy. Re-ranking every document instead of vector search would not scale, since it costs as much as a full text search, so retrieval stays two-step.

• Vector search is the cheap first step across millions of documents; the re-ranker is the expensive second step on a short list.

▸In their words

“the accuracy gains are usually worth the latency loss.” Gaurav · at 6:10 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-09-28). Read the whole episode summary →

▸More from this episode