Why is an LLM chatbot slow, according to Gaurav Sen?

In his mock AI interview, Gaurav Sen says a slow retrieval-augmented chatbot can be stuck in one of three places: ingesting uploaded documents, fetching chunks from the vector database, or the language model generating its response. Once the model is confirmed fast and no documents are being uploaded, he says the usual culprit is the vector database search.

• Gaurav Sen also asks whether the system uses re-ranking or query rewriting, since both add steps to each request.
• On the model side, he says slowness can come from a streaming issue or from the wrong model parameters.

▸In their words

“Usually the culprit here is the vector DB.” Gaurav Sen · at 2:19 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-10-05). Read the whole episode summary →

▸More from this episode