How does prompt caching work in LLMs, according to Gaurav Sen?

Gaurav Sen describes the basic version as a key-value cache placed between your system and the LLM. In his travel-planner example, the first query for a Bangalore-to-Mysore trip is a cache miss, so the system calls the LLM and stores its plan. When another user sends the exact same text, for instance by picking the same form fields, the stored response is returned without calling the model again.

• Exact-match caching only works when users send identical text, such as values chosen from a form.

Source: our own transcription of the episode. The full conversation is on Gaurav Sen (2026-08-03). Read the whole episode summary →

▸More from this episode