Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A clear, short explanation of why a RAG pipeline with too many retrieved chunks loses accuracy and how a re-ranker fixes it.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“For a human we know not is very important, but a vector may not be able to encapsulate that.” Gaurav · at 3:59 — open in the transcript →
“the accuracy gains are usually worth the latency loss.” Gaurav · at 6:10 — open in the transcript →
In this seven-minute mock interview on Gaurav Sen's channel, the candidate, Gaurav, is told that a team's RAG chatbot is not accurate enough. The offline indexing side has been checked and works, and the team does no query rewriting, so the diagnosis narrows to online retrieval. The team pulls the top 20 to 25 chunks per query, and Gaurav argues that this floods the context window, turning the model's job into a needle-in-a-haystack search that lowers quality and invites hallucination.
The proposed fix keeps the wide retrieval but adds a re-ranking and filtering step. A cross-encoder re-ranker scores each chunk's tokens directly against the query's tokens, catching distinctions that embeddings blur, such as negation, numeric price ranges or a seven-day refund window, and only the top 10 or so chunks reach the model.
The interviewer then probes the trade-offs. Re-ranking everything would cost as much as a full text search, so vector search stays as the cheap first pass. The extra 50 to 100 milliseconds is worth it, Gaurav says, if faithfulness and answer relevancy rise by roughly 5 to 10 percent.
In the mock interview on Gaurav Sen's channel, the candidate Gaurav rules out offline indexing and query rewriting, and calls a poor embedding model unlikely. The likely cause is retrieval of chunks marked relevant that are not really relevant. The team takes the top 20 to 25 chunks per query, which fills the context window, so the LLM must search a haystack for the answer and becomes more likely to hallucinate.
Gaurav, the candidate in Gaurav Sen's AI interview #7, says 20 to 25 chunks is too many to pass to the model because it pollutes the context. Retrieving 20 to 25 from the vector database is fine, but a re-ranking and filtering step should cut that down to the 5 to 10 most relevant chunks, with 10 as his working number, before they reach the LLM.
In Gaurav Sen's mock AI interview, Gaurav explains that a re-ranker is a cross-encoder that skips embeddings. It takes the user query's tokens and each chunk's tokens together and returns a similarity score per chunk. With 20 to 25 retrieved chunks you get 20 to 25 scores, keep the highest, drop the rest, and pass the survivors to the LLM, which should raise accuracy and reduce hallucination.
Gaurav, in Gaurav Sen's AI interview #7, says embedding similarity blurs meanings that matter to a human. A laptop priced $300 to $400 is a range that is hard to embed. Sentences that differ only by 'not', such as wanting or not wanting a seven-day refund period, land close together, as do an aircraft carrier and an aircraft part. Token-level matching through a re-ranker handles these better.
Gaurav's answer in Gaurav Sen's mock interview is usually yes. Re-ranking 20 to 25 chunks adds about 50 to 100 milliseconds, which he considers worth it for a roughly 5 to 10 percent accuracy bump measured on metrics such as faithfulness and answer relevancy. Re-ranking every document instead of vector search would not scale, since it costs as much as a full text search, so retrieval stays two-step.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00We have a RAG architecture, we have a RAG pipeline, but in that architecture we are not getting that level of accuracy. So what changes can you make or what updates can you do in that architecture to make it more accurate? By accurate you mean it should be correct in terms of the answers it's giving? Not just correct, there are several metrics but I wanted to focus first on the architecture. Okay, okay. There are a few components in RAG, maybe I can draw it out and then I can double down. Yeah, for sure. Let's take it on the whiteboard then. In any RAG pipeline with this kind of an architecture, some things could go wrong. So if you're seeing that there's a lack of accuracy like you mentioned, then
0:34there are some usual suspects. Okay, one of the things is if there's a user query which is coming to your system, are we picking up the most relevant chunks? Yeah, that is actually what we are facing the problem with we have checked the offline thing okay it is working fine but we are we are having some issues with the online process okay okay so this part is fine let's go for the online process here things could only go wrong in some parts of the system one is that are you doing any kind of query rewriting when you're getting a so are you taking the query and then writing it in your own way? No, for our use case, it is not required. Okay. Okay. So then here the embedding model could be poor, but chances low.
1:13So most likely what's happening is you're getting chunks, which is being marked as relevant when compared to an input query from the user, but maybe they're not very relevant. So can you tell me how many chunks you're picking up in Topk? We decided on a number of 20 to 25. Yeah, 20 is the number I guess. 20 to 25 is quite a large number of chunks because that can be polluting the context. If you have a lot of chunks, then one of the things which happens is you fill up the context window and LLM then has to search in the context. It becomes a needle in a his stack problem. Its performance is expected to reduce. It can also hallucinate. Most likely what's happening is because you're picking 20 to 25 chunks,
1:53it's likely that many of these are relevant and they are causing the issue. So what we can do here is reduce the number of chunks to five or 10, let's say 10. It's okay to pick up 20, 25 chunks. Why did you pick up 20-25 chunks though? Just empirically, we decided upon, you know, there are a lot of chunks that we are creating based on the chunk size. So we decided that top 20 chunks would suffice, I guess. Okay, okay. Maybe what has also potentially happened is the team found that they are not able to get the relevant chunks, so they are digging deeper and deeper. And what's happened is the context is polluted. This is usually something which happens if they are to get the relevant chunks. So they are digging deeper and deeper. And what's happened is the context is polluted.
2:25This is usually something which happens if they're not getting the relevant chunks. So, okay, let's take 20 to 25. So that the relevant chunks surface after this, what you want to do is something called re-ranking and filtering. Okay. You take the best, most relevant chunks and you filter it down to 10 from 20 to 25. Re-ranking basically is taking the chunks that you have and checking them again for similarity with the user query. Now the vector database is doing this, it's taking the user query converting it into an embedding let's call it v1 and it's taking the vector database embeddings. Let's say V100, V101, V102 and so on. Okay. Thousands and thousands of embeddings. And it's finding out which vector is most similar to this one or which top 20
3:18vectors are closest to this one. The problem with this is if there are queries like find a laptop within the range, $300 to $400, then numerical things like that ranges are very hard to embed. That kind of meaning is very hard to embed. Or if you say this is not what I want versus this is what I want. Both of them are going to come very close to each other in terms of embedding. Or if you have the example, this is an aircraft carrier and that this is an aircraft part, then both of them can be potentially very close to each other in terms of embedding. Or if you have the example, this is an aircraft carrier and that this is an aircraft part, then both of them can be potentially very close to each other also.
3:49Or if you say that I don't want anything with a refund period of seven days versus I want something with a refund period of seven days are going to have very similar vectors because the meanings are rather close to each other. For a human we know not is very important, but a vector may not be able to encapsulate that. So what we want to do is we don't want embedding level similarity. This is not what we want. We instead want token level, similarity or matching. This is what we need. And for this, you need something called a re-ranker internally. It's a cross encoder. So what this does is it takes the user query and doesn't convert that into an embedding. It directly takes the tokens from the user query and it takes the tokens from
4:34every chunk and passes this to something called a re-ranker. Like I said, it's a cross encoder internally. It gives you a score. Okay. So for every chunk and the user query you're getting a score the ones with the highest similarity scores rest of the 10 or 15 will be dropped once you have those chunks with the highest similarity scores that can be passed to the LM and now the accuracy scores are expected to improve also the chance of hallucination usually reduces okay so is it happening for every chunk or is it happening for a bunch of chunks together it happens for every chunk individually a chunk may for a bunch of chunks together? It happens for every chunk individually. A chunk may have, let's say, 800 characters or maybe 200, therefore, words.
5:11So 200 words are going to be compared with the input user query. Every single chunk is going to be compared with every single user query. I mean, with one user query only. And we'll be getting a score. You'll be getting a score. You'll be getting 20 to 25 scores. For every chunk, you'll be getting a score with respect to this. score chunks will be picked up so if re-ranker is that good why don't we use it directly like why even the process of you know going through to the vector db and all of that okay one thing is that this does not scale very well so this is a two-step process because the first step is cheap the second one is a bit expensive converting into embedding and then finding amongst millions of documents is possible but doing a re-ranker which is token
5:49level checking with every single document chunk is going to be as expensive as doing a text search itself so that's the reason why we don't do this and what about the latency that it is introducing what are the trade-offs basically if i am like what kind of accuracy will make me introduce a re-ranker okay i mean the latency for 20 to 25 chunks is there. It becomes an additional part of your process but the accuracy gains are usually worth the latency loss. So if we are thinking about this, if it takes like 50 milliseconds or 100 milliseconds and if you're okay giving that for your accuracy bump, then yes, a re-ranker is a good idea. Roughly 5 to 10% accuracy bump then then yes a re-ranker is a good idea roughly
6:25five to ten percent accuracy bump is what we are looking at we have some specific metrics for this faithfulness answer relevancy so if those numbers are going up then yes the increase in latency and the increase in cost also is then justified okay cool thank you so much gaurav i think you are the one who we need to hire. Yes. This is AI. And agentic rag and also agentic evals. So can you introduce yourself a bit? I'm Gaurav. I have 10 years of experience in this industry working apps. And in the last two years, I have been working on some AI, on an AI startup. Two years of experience. Yes.