Technology
AI Interview #8 - Why is the chatbot so slow?
Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A compact walk-through of how to locate the slow stage in a RAG chatbot before trying to fix it.
Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A compact walk-through of how to locate the slow stage in a RAG chatbot before trying to fix it.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“Usually the culprit here is the vector DB.” Gaurav Sen · at 2:19 — open in the transcript →
In this short role-play from Gaurav Sen's AI interview series, Gaurav plays a candidate asked to make a company's chatbot respond in milliseconds. Rather than guessing, he first narrows where the delay can live: a retrieval-augmented generation (RAG) system can be slow while ingesting documents, while fetching chunks from the vector database, or while the language model generates its reply.
With the model ruled out and uploads absent, Gaurav names vector search as the usual culprit. A million documents split into five to ten chunks each means roughly 10 million vectors per query, so he proposes tuning the approximate nearest neighbour indexes such as HNSW and IVF, running searches in parallel, pooling database connections, caching common queries in Redis and sharding the database by query type.
When the interviewer switches to slow uploads, Gaurav walks through parsing, chunking, embedding and storage, and singles out semantic chunking as the likely bottleneck at scale because it repeatedly scores similarity between chunks. The clip ends on a joke about a 4 LPA salary.
In his mock AI interview, Gaurav Sen says a slow retrieval-augmented chatbot can be stuck in one of three places: ingesting uploaded documents, fetching chunks from the vector database, or the language model generating its response. Once the model is confirmed fast and no documents are being uploaded, he says the usual culprit is the vector database search.
Gaurav Sen suggests tuning the approximate nearest neighbour index, such as HNSW or IVF, then running the search in parallel and pooling database connections. He adds a Redis cache with vector search for common queries, horizontal scaling, and sharding the vector database so each query is routed to the shard that matches its type.
Gaurav Sen says a semantic chunker splits documents and then computes similarity scores between the chunks, repeating that work many times. That makes it slow when there are many documents, so although it sounds good in theory, he says it does not hold up in practice at scale and is the likely bottleneck when uploads are slow.
Gaurav Sen says uploads pass through parsing, chunking, conversion into vector embeddings and storage in the vector database. Parsing can be slow and also inaccurate, because the format of a user's document is not known in advance, and chunking is the other common slow step, especially when a semantic chunker is used.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00I'm Gaurav and I have been working on AI applications for the past two years but my knowledge... Can you make chatbot fast because our chatbot is very slow our customers are complaining that it is very slow so I want to make it fast like just milliseconds. Milliseconds is difficult there's an external I'm not sure if you're hosting your own LLM if there's an external API call that you're making then it depends also on that external system. But I'll try to make it as fast as possible. Can I go through some of the things that we should? Yeah, for sure. Let me also call the technical person too. To evaluate? Yeah. Okay. Okay. Sure. Sure. Okay. So we know that the chatbot being used here is slow.
0:40I just want to pinpoint where is the problem. So firstly, could you tell me, is it rag based or is it agentic it's a rag-based but it's a rag-based but it uses okay company documents to yeah we have a personal documents yeah okay in rag-based there are two parts to the chat bot one is that if a user is uploading documents if they are uploading an invoice or uploading some sort of their profile history versus or something related to the query so is there any document upload or is it like being answered only using company documents it's only inference based so they are just using the bottom they are not uploading anything okay so okay so I know that document upload is not there, which means chunking converting into vectors is not taking a long time.
1:32What is happening is existing inference. Okay. So here there are two parts to it. One is our system is taking time to fetch documents from the vector DB. The other is the LLM is taking a long time to give a response. If the LLM is taking a long time, it could be a streaming issue or it could be that we are just not using the right parameters with the LLM. Could you tell me which side is it? And also, are you using any kind of re-ranking or query rewriting anywhere? We're not using query rewriting. It's just we have one re-ranker and there is no LLM issue as well. Okay, the LLM is fast. Okay, so the only thing that could be taking a long time in my opinion, usually the culprit here is the vector DB.
2:22So the search is slow. So you have a user query. usually the culprit here is the vector DB. So the search is slow. So, you have a user query and it's taking a long time to fetch the chunks from the vector database. Here, most likely what has happened is the algorithm, I'm not very sure which one you're using. That is hierarchical navigable small world. There is IVF. If these are not tuned right, then finding the vectors takes a long time. This approximate neighbor search takes a long time. The user query after being converted into an embedding takes time to actually find the nearest chunks. So coming back to the original question, how large is this vector database? It's quite large. It is more than like we had a more than million documents actually.
3:02Initially. Okay. So, okay. So if you have a million documents and each of those documents may be broken into five to let's say ten chunks? Yeah, okay so we have something like 10 million vectors that need to be searched for every single user query. This is going to take time whether we optimize also on ANN. So one of the things that we can work on this, but the second thing is we want to probably run these algorithms in parallel. So the ANN algorithm should be running in parallel. Also I'm assuming that the user queries are being accepted in parallel, but if this is taking time, there is a way to do database connection pooling also. Maybe you want to cache some of these documents, the most recent ones in your system. So for the cache,
3:53you can use Redis which allows vector search that will reduce the time to search for common user queries. Apart from this, generally horizontally scaling the vector database is one thing you can do. Depending on what kind of user query is coming or what kind of documents you have, you can split the database into multiple shards. And so the system can query the right shard, can route the query to the right shard, depending on the kind of user query which has come. So Gaurav, what if the document upload is taking time? What do you think? Where should we optimize that? Okay, so if it was the document upload, there's? What do you think? Where should we optimize that? Okay. So if it was the document upload, there's a few other things which are happening before this step.
4:30The document upload needs some sort of chunking of the original document, which is being uploaded by the user. Then it has to be before chunking, actually we have to pass the document. This is going to extract the data. This could be slow. This could be one potential part where things are slow. It can also be a little inaccurate also because a user document may not be passed the right way. We may not know the format beforehand. After which we go for chunking. Then each of these chunks is going to be converted into vector embedding. This vector can now be stored in the vector database. So one part of this is usually slow. Parsing, I don't know what library you're using. Chunking, usually if there is a problem here,
5:14there's something called a semantic chunker, which sounds great in theory, but doesn't really work in practice when you're looking at scale. So is this something you're using? Yeah, we are actually using a semantic chunker. Okay. Most likely then the semantic chunker is the one which is taking time. The reason for that is because when you're using a semantic chunker, you are chunking the documents and then finding similarity scores between each chunk. And this happens multiple times, so it is quite slow, especially if you have lots and lots of documents. Okay, now done. Now you get back to work and do all of this i am dumb you know no no you can be smart also wow just want to make it fast so we have to make everything faster otherwise there is no logic
6:02right sir what is the salary again that you said? Salary is 4 LPA. I will definitely leave fast sir.