What is a semantic cache for LLM prompts, as Gaurav Sen explains it?

A semantic cache, in Gaurav Sen's explanation, handles prompts that mean the same thing but use different words. Instead of keying on the query text, it stores a vector embedding of each past query. A new query is embedded, compared against the stored vectors, and if the nearest one scores above a threshold such as 0.9, its cached response is returned.

• Cosine similarity is the popular, fast way to score closeness, according to Gaurav Sen.
• For caching, only the single closest match is used, not a top-3 or top-5 list.

▸In their words

“We need something called a semantic cache.” Gaurav Sen · at 2:39 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-03). Read the whole episode summary →

▸More from this episode