Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A quick, interview-ready explanation of when exact caching works, when a semantic cache is needed, and what it costs in accuracy and latency.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“We need something called a semantic cache.” Gaurav Sen · at 2:39 — open in the transcript →
“latency definitely is an issue because vector search is slow.” Gaurav Sen · at 4:41 — open in the transcript →
In this six-minute mock interview on Gaurav Sen's channel, an interviewer asks Gaurav how to cache prompts sent to an LLM, using a travel planner that answers queries like a trip from Bangalore to Mysore.
Gaurav separates two cases. When users send identical text, such as a value picked from a form, a plain key-value cache sits between the system and the model: the first query misses and calls the LLM, and later identical queries reuse the stored plan. Stale answers, like a road strike, are handled with a short timeout or least-recently-used eviction. When prompts mean the same thing in different words, a semantic cache stores query embeddings as keys and returns the nearest match above a similarity score such as 0.9, often measured with cosine similarity.
The semantic version has costs. Embeddings lose detail, so requests for a $1,000 and a $3,000 laptop can look alike and get the wrong cached answer. Approximate nearest-neighbor search is also far slower than a hash lookup. Gaurav still sees it as cheaper than a full LLM call, because an embedding model is fast and inexpensive to run.
Gaurav Sen describes the basic version as a key-value cache placed between your system and the LLM. In his travel-planner example, the first query for a Bangalore-to-Mysore trip is a cache miss, so the system calls the LLM and stores its plan. When another user sends the exact same text, for instance by picking the same form fields, the stored response is returned without calling the model again.
Gaurav Sen gives two options. If the answer can go out of date quickly, as when a strike blocks the road between Bangalore and Mysore, each cached response gets a timeout, for example one minute. If staleness is unlikely, the cache can instead use least-recently-used (LRU) eviction and drop old prompts as space is needed.
A semantic cache, in Gaurav Sen's explanation, handles prompts that mean the same thing but use different words. Instead of keying on the query text, it stores a vector embedding of each past query. A new query is embedded, compared against the stored vectors, and if the nearest one scores above a threshold such as 0.9, its cached response is returned.
Gaurav Sen says a semantic cache adds points of failure. Embedding a query loses information, so prompts asking for a $1,000 laptop and a $3,000 laptop can look nearly identical and get the same inexact answer. It also adds latency, because approximate nearest-neighbor vector search is much slower than an exact key lookup in a hash map.
Yes, in Gaurav Sen's answer. Checking a semantic cache needs only an embedding model, which is fast and cheap to run. A real LLM call runs the prompt through all of the model's attention blocks, and since most teams use a hosted provider rather than running the model themselves, that call usually costs more.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Yeah, Gaurav. So we have discussed, you know, the basic agent, whatore to Mysore and then I am going for a vacation to Mysore I stay in Bangalore is it like this or are they exactly like form clicks and then the exact same text is being sent to the LM for this example let's take that their exact text okay so starting with the most basic use case where there is exact text that is being sent by the user to our system. If we introduce a cache here between the LLM and our system, the first time the user sends us a query, let's say I'm going to Mysore from Bangalore, so that's the query Mysore to Bangalore. The first time we'll have a cache miss, we'll go to the LM, say that this is the user query.
1:08The LM is going to give us a plan how to reach Mysore and which hotels to book. That will be the response. The next time another person has the same query, let's say they selected the same field on the form, this new user is going to go to our system and since we populated the cache before responding or after responding this cache is now populated with this kind of query Mysore and Bangalore and when this key is found the same response can be picked up there is one problem though it's possible that the response has become stale let's say there's a strike in between the road to Mysore and Bangalore so in those cases how do I make this system get rid of stale responses? Either I can keep a timeout, like the response is valid
1:50only for one minute. Or if I know that this is not really going to happen, then I can keep it with a typical LRU fashion also, at least recently used, I can kick out prompts as needed. So this is a very basic way to cache things. So Gaurav, we understood, you know, when we have same text, basically, not even similar, same text. So how do we do caching? Let's say now your prompts are basically semantically similar. So they have the same meaning, but it is not exact text. So how are you going to do caching in that case? Someone says, I stay in Bangalore. I'm going to Mysore. And someone says, I'm going to Mysore and someone says I'm going to Mysore from Bangalore on the way I want to do this.
2:31So they are both semantically the same. Correct. Although the text is not going to match in the cache. So here we need a different kind of cache. We need something called a semantic cache. And internally a semantic cache is something that does comparisons based on vectors. Typically a cache is a key value cache where you have the user query, you check amongst the keys which one is an exact match and you get 1 and 0 everywhere else. So it's exact match. But with a semantic cache you get a user query you have the keys and values the key here is not the ID of the query but the vector value okay this is like a mathematical representation of what Bengaluru to Mysore looks like and then the query
3:17tries to actually find the nearest vector amongst all these vectors it might pick up the top five top two top 3 but since it's caching so we are in the top one, the closest one to our incoming query, that value can be picked up to give a response. The problem with this though is that earlier you had exact match, now you're always going to have an approximate match. And so if the query is converted into a query vector, a mathematical representation of it, and the previous queries are converted into vectors, if these two queries are close to each other semantically, then we assume that this similarity score is going to be high. There are many ways to calculate similarity. Cosine similarity is really popular fast. Google has its own
4:04thing. But if the similarity score let's say is really popular fast. Google has its own thing. But if the similarity score, let's say, is greater than 0.9, then we say that this is the exact query that people had asked earlier. And so now we can give a response. So basically, what is the trade-off here? Let's say if I introduce semantic cache, what is the trade-off that I am basically making? We are introducing more points of failure, actually. One is that when you are embedding a query and you are embedding this query you are losing some information. So they may look similar like give me a laptop with cost $3,000 and give me a laptop which is $1,000. Those two vectors may be very similar they might come up ranking in the cache as high similarity. So there is
4:41a problem that you are going to give inexact answers. What about latency and cost? If the cache is going to be checked every time, cost is not that bigger issue but latency definitely is an issue because vector search is slow. What we try to do is approximate nearest neighbors is the kind of search we run. There's various algorithms here but this is much slower than just doing a exact key comparison exact key comparison is literally like a hash map this is not a hash map this is like a nd architecture okay so basically we are you know comparing increasing latency yeah increasing latency but do you think when we are talking about cost here also you are using an llm and if i don't introduce a cache and i directly do a LLM call that is also a
5:25single LLM call so what is the difference between the cost there so an LLM the cost is going to be usually higher because here you have an embedding model which is really fast it's cheap to run but if you make an LLM call then it's going to actually go through the entire set of attention blocks over there. If most likely you're not going to host it yourself, you're going to have it through some provider that is going to cost you more. Cool. Yeah. Thanks Gaurav. It was a good answer. If I ask you, so you have too many people coming for your like using your application, right? Although it won't happen, but let's say for example, like you have,