Gaurav Sen

AI Interview #9 - Why is the Agent so slow?

⏱ 7 min listen · ~2 min read 🎙 Tanishk

A compact walkthrough of the questions to ask and the first fixes to try when an LLM agent with several tools feels slow.

Key takeaways

01Tanishk's slow agent is a single-agent performance-review app calling a hosted LLM over API, with about five MCP tools for employee data and git commits.
02Synchronous tool calls make each of the five tools wait for the previous response, which Tanishk names as the first cause of the agent's latency.
03An MCP server may expose 10 tools when a department needs 5, and Tanishk says filtering them cuts both latency and context bloat.
04Calling a slow employee-data tool can be skipped by caching stable facts such as leave balances and passing them in the system prompt, per Tanishk.

Chapters

0:00 · Scoping the slow performance-review agent1:42 · Synchronous tool calls as the first bottleneck2:56 · Filtering MCP tools to cut latency and context3:56 · Caching employee data instead of calling a tool5:05 · Merging overlapping tools and the case against batching

Read the full summary →

AI Interview #8 - Why is the chatbot so slow?

⏱ 6 min listen · ~1 min read

A compact walk-through of how to locate the slow stage in a RAG chatbot before trying to fix it.

Key takeaways

01Gaurav Sen's mock interview opens with a client who wants a slow company RAG chatbot to answer in milliseconds.
02Gaurav Sen splits RAG latency into three suspects: document ingestion, vector database retrieval and the language model's response.
03A million documents cut into five to ten chunks each leaves about 10 million vectors to search on every user query, Gaurav Sen estimates.
04Semantic chunking sounds great in theory but slows uploads at scale, Gaurav Sen argues, because it keeps scoring similarity between chunks.

Chapters

0:00 · A client demands a millisecond chatbot0:39 · Narrowing the slowdown inside a RAG pipeline2:19 · Vector search as the usual bottleneck2:57 · Parallel search, caching and sharding at 10M vectors4:19 · Upload pipeline and the semantic chunker cost

Read the full summary →

AI Interview #7 - Why is this chatbot failing?

⏱ 7 min listen · ~2 min read

A clear, short explanation of why a RAG pipeline with too many retrieved chunks loses accuracy and how a re-ranker fixes it.

Key takeaways

01Gaurav's mock-interview chatbot is a RAG pipeline whose offline indexing works but whose online retrieval returns answers that are not accurate enough.
02Pulling the top 20 to 25 chunks per query pollutes the context window, Gaurav says, making the LLM hunt for a needle in a haystack and hallucinate.
03A cross-encoder re-ranker compares query tokens with each chunk's tokens and keeps about 10 of the 20 to 25 chunks retrieved by vector search.
04Re-ranking every document would cost as much as a full text search, so Gaurav keeps vector search as the cheap first step before the re-ranker.

Chapters

0:00 · Scoping an inaccurate RAG chatbot0:55 · Online retrieval and the oversized top-k2:34 · Re-ranking and filtering the retrieved chunks3:20 · Where embedding similarity breaks down5:27 · Two-step retrieval cost and latency trade-offs

Read the full summary →

Is system design still relevant in 2026?

⏱ 12 min listen · ~2 min read 🎙 Arvind

It gives working engineers a short, practical answer to whether AI makes their design and coding experience obsolete, and what to learn next.

Key takeaways

01Gaurav Sen and Arvind of the Gen Academy ask how software engineers can stay relevant through the next five to ten years of AI tools.
02Arvind's first step is AI literacy: learning what an LLM is and what context window, temperature, top p and RAG mean before building anything.
03Gaurav Sen says entries to his India AI hackathon, aimed at areas with one doctor per 11,000 patients, all looked the same after AI help.
04Companies that gave every engineer Cursor now generate so much code, Arvind says, that they cannot tell what is safe to promote to production.

Chapters

0:00 · Staying relevant as an engineer in the AI era2:25 · AI literacy from context windows to RAG4:32 · Judging AI-generated code from Cursor5:24 · The India AI hackathon and look-alike entries6:39 · Design and decision work moving upstream8:39 · Finding a path through an unmapped field+1 more

Read the full summary →

AI Interview #6 - 1M context window in LLM?

⏱ 7 min listen · ~1 min read 🎙 Tanishka

A fast, whiteboard-style walkthrough of a basic RAG pipeline and the first fix for PDFs full of tables.

Key takeaways

01Tanishka's scenario is an internal support bot with 100,000 users whose growing pile of company documents no longer fits in the LLM's context window.
02Tanishka's RAG pipeline cuts documents into 800 to 1,000 character chunks, embeds them, and hands the top 5 or 10 matches to the LLM per query.
03Postgres indexes fall short for this, Tanishka argues, because each chunk becomes a vector of 768 and up to 8,192 dimensions rather than one value.
04Scanned PDFs and screenshots of tables defeat parsers like PyPDF and PyMuPDF, so Tanishka adds layout detection to the ingestion pipeline.

Chapters

0:00 · A support bot outgrowing its context window0:22 · Chunking company documents into embeddings1:05 · Vector database versus Postgres indexes1:56 · Query retrieval, top-k chunks and grounded answers4:38 · Layout detection for PDF tables and screenshots

Read the full summary →

AI Interview Questions #5 - Choose RAG or Agents?

⏱ 4 min listen · ~1 min read

In four minutes it gives a crisp, interview-ready answer to the RAG versus agents question using one concrete travel-booking example.

Key takeaways

01Gaurav Sen's mock interview question is whether a flight, hotel and trip-planning app should use RAG or an AI agent.
02RAG, in Gaurav Sen's description, embeds the query, pulls similar chunks from a vector database and adds them to the LLM's context.
03Gaurav Sen names three agent traits RAG lacks: a context-gathering loop, tool calls that take real actions, and access to live data.
04Flight and hotel data is the catch for RAG, says Gaurav Sen: you do not own it, cannot store it, and a retry loop does not fix that.

Chapters

0:00 · A travel-booking app as the interview prompt0:36 · How retrieval-augmented generation differs from agents1:45 · Static owned data versus live flight and hotel data2:54 · Agentic RAG and the closing whiteboard joke

Read the full summary →

Research Paper Walkthrough: RAGAS Evals

⏱ 39 min listen · ~2 min read 🎙 Tanishq

It turns a widely cited evaluation paper into a practical recipe for judging a RAG pipeline with three yes-or-no LLM checks.

Key takeaways

01Gaurav Sen and Tanishq walk through RAGAS, a five-page paper with about 2,100 citations on scoring RAG pipelines without reference answers.
02RAGAS rates a RAG system on faithfulness, answer relevance and context relevance, each built from the question, retrieved context and answer.
03Tanishq's setup runs three single-purpose LLM judges, one per RAGAS metric, and a query passes only when all three return yes.
04Tanishq warns that generic reference-free metrics such as RAGAS can show false success rates unless judge prompts are tuned to the use case.

Chapters

0:00 · Reading the RAGAS abstract and its three problems5:26 · Related work on LLM judges and reference scoring6:46 · The standard RAG setting of question, context and answer9:56 · Faithfulness, grounding and citations11:51 · Answer relevance and responses that drift13:00 · Context relevance as precision versus recall+4 more

Read the full summary →

AI Interview Questions #4 - Evals in AI

⏱ 11 min listen · ~2 min read 🎙 Tanishq

A short, whiteboard-style answer to the interview question of how you would prove an LLM app is reliable enough for healthcare.

Key takeaways

01Tanishq's mock-interview brief is a hospital app on Gaurav Sen's channel that must use AI without ever giving patients a wrong answer.
02Agent traces record system prompt changes, tool calls, tool responses and model decisions, and Tanishq treats them as the data evals run on.
03Tanishq's three starter eval metrics are answer relevancy, groundedness and tool sequence accuracy, with completeness and correctness as extras.
04Ground truth is often missing, so Tanishq builds a synthetic dataset or a golden dataset with a subject-matter expert before evaluating an agent.

Chapters

0:00 · A hospital that cannot afford wrong answers0:59 · Reliability as observability plus evals2:51 · Evals as the QA layer for AI agents4:50 · Answer relevancy, groundedness and tool order6:41 · Eval ownership and golden test datasets8:14 · Inputs needed to compute each eval metric

Read the full summary →

AI Interview Questions #3 - Prompt Caching with LLMs

⏱ 6 min listen · ~1 min read

A quick, interview-ready explanation of when exact caching works, when a semantic cache is needed, and what it costs in accuracy and latency.

Key takeaways

01Gaurav Sen's travel-planner example caches a Bangalore-to-Mysore query so the next identical request skips the LLM call entirely.
02A semantic cache, as Gaurav Sen describes it, keys stored answers by query vectors and returns the single nearest match instead of an exact key hit.
03Gaurav Sen treats a similarity score above 0.9, often cosine similarity, as close enough to serve the cached response.
04Embeddings can blur a $1,000 laptop request and a $3,000 one, so Gaurav Sen warns a semantic cache may return inexact answers.

Chapters

0:00 · Exact-match caching for a travel-planner query1:34 · Clearing stale answers with timeouts or LRU2:06 · Semantic caching for prompts with the same meaning4:14 · Accuracy and latency trade-offs of a semantic cache5:15 · Embedding model cost versus a full LLM call

Read the full summary →

AI Interview #2 - Context in LLMs with @aloo_explains

⏱ 8 min listen · ~1 min read 🎙 Tanishq

A short, whiteboard-style answer to a common AI interview question: what an LLM's context contains and where memory fits in.

Key takeaways

01Tanishq defines an LLM agent's context as the system prompt, tool descriptions, user query and every tool response gathered so far.
02Tanishq's Japan trip example shows context growing step by step as hotel and then flight tool outputs are added to it.
03For context running to millions of tokens, Tanishq proposes storing five separate summaries in a memory layer for five travellers.
04A multi-agent split divides context between planning and summarisation agents, a separation of concerns Gaurav Sen compares to microservices.

Chapters

0:00 · Candidate background and the context question0:26 · Tracing context through a Japan trip planner2:46 · Summaries and a memory layer for huge contexts4:08 · Multi-agent designs as a way to split context4:37 · How memory differs from context, Redis to Postgres

Read the full summary →

AI Interview #9 - Why is the Agent so slow?

AI Interview #8 - Why is the chatbot so slow?

AI Interview #7 - Why is this chatbot failing?

Is system design still relevant in 2026?

AI Interview #6 - 1M context window in LLM?

AI Interview Questions #5 - Choose RAG or Agents?

Research Paper Walkthrough: RAGAS Evals

AI Interview Questions #4 - Evals in AI

AI Interview Questions #3 - Prompt Caching with LLMs

AI Interview #2 - Context in LLMs with @aloo_explains

▸Questions this show answers