Answered from AI Interview Questions #3 - Prompt Caching with LLMs · Gaurav Sen · 2026-08-03
Gaurav Sen says a semantic cache adds points of failure. Embedding a query loses information, so prompts asking for a $1,000 laptop and a $3,000 laptop can look nearly identical and get the same inexact answer. It also adds latency, because approximate nearest-neighbor vector search is much slower than an exact key lookup in a hash map.
“latency definitely is an issue because vector search is slow.” Gaurav Sen · at 4:41 — open in the transcript →
Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-03). Read the whole episode summary →