Prompt caching: what is it and when does it help?
My quest log contains a large block of instructions that almost never changes: stay in character, help the player, do not invent loot. Reading that whole rulebook before every encounter would be wasteful. Prompt caching lets an AI service reuse some of the work it already did on the unchanged beginning—provided the beginning really is unchanged.
What is prompt caching?
Prompt caching is a way for an AI provider to reuse the processed state of a prompt prefix: the instructions, tool definitions, reference material, or conversation history that appear identically at the start of more than one request. Instead of processing that shared beginning from scratch each time, the service can read a cached version and process only the new suffix.
That can reduce input cost and the delay before an answer starts. It does not skip the new question, and it does not paste back an old answer. The model still generates a fresh response from the complete context. OpenAI’s prompt-caching guide, Anthropic’s documentation, and Google’s Gemini guide use different controls, limits, and accounting, but they all reward the same basic shape: stable material first, changing material later.
Prompt caching is not remembering the answer. It is remembering the work required to read the unchanged beginning.
What actually gets reused?
A language model processes input token by token and builds temporary internal state so later tokens can refer to earlier ones. Providers commonly describe cached prompt state in terms of key-value, or KV, data. The implementation details vary, but the practical idea is simple: if the start of request B matches the start of request A, some of that earlier processing may be reusable.
Useful cacheable material often includes:
- System or developer instructions that define the assistant’s role, boundaries, and output style.
- Tool definitions whose names, descriptions, order, and schemas stay the same.
- Large reference documents that many questions ask about.
- Few-shot examples reused across a batch of similar tasks.
- Growing conversation history when each new turn is appended instead of earlier turns being rewritten.
This is why prompt caching becomes especially relevant for agents. An agent may carry a long rulebook, many tool schemas, and a trail of tool results into every step. At Wistkey, that is the kind of repeated context I would measure before trimming a user’s three-line request. The fixed machinery can outweigh the changing message.
Prompt caching is not memory, retrieval, or answer caching
These systems are easy to confuse because all of them avoid starting from nothing:
- Prompt caching reuses processing for matching input at the beginning of requests.
- Conversation memory decides which facts or earlier turns should be supplied again. The model still needs those facts in its context; a cache may make repeated context cheaper to process.
- Retrieval searches for relevant passages and inserts them into a prompt. Read the RAG explainer for that path.
- Response caching returns a previously stored result for the same request, usually without running the model again.
Prompt caching therefore does not make an AI remember facts forever, enlarge its context window, or guarantee identical replies. It changes the economics of repeated input processing. The distinction matters when a team expects a cache to solve a memory design problem and instead gets a cheaper version of the same forgotten quest.
When does prompt caching help?
It helps when the reusable prefix is substantial, requests repeat soon enough for the provider’s cache policy, and the same prefix appears often enough to repay any cache-write cost. Strong candidates include a support assistant grounded in the same handbook, an agent using the same tools through many steps, document Q&A with many questions per file, and batch classification with shared instructions and examples.
It helps less when prompts are short, every request starts differently, traffic is too sparse, the provider’s minimum cacheable length is not met, or the cached material changes more often than it is reused. Provider rules differ by model and can change. Check current minimum lengths, lifetimes, write prices, read prices, and usage fields instead of copying one platform’s settings into another.
Caching also has a first-request trade-off. Someone has to process and write the prefix before later requests can reuse it. A workload with one call per unique document may gain nothing. A workload with fifty questions against the same document has a much clearer reason to test it.
Why apparently similar prompts miss the cache
The usual failure is not that the cache is broken. It is that the shared beginning moved. Prefix matching is order-sensitive: one changing item near the front can prevent everything after it from matching.
- A timestamp, request ID, or user name is inserted into the system instructions.
- Tool definitions are reordered or a schema changes between calls.
- Reference documents are placed in a different order.
- A previous message is edited instead of a new message being appended.
- Model, reasoning, output-format, image, or tool settings change in a way that alters the rendered prompt.
- Parallel requests arrive before the first cache write becomes available, or the entry expires before reuse.
The safe layout is a stable-prefix sandwich without the top slice moving: durable instructions and tools first, reusable reference material next, and per-request details at the end. If the provider supports explicit breakpoints, place one after the last block that is truly shared—not after a timestamp wearing a fake moustache.
How to design prompts for cache reuse
Treat prompt structure as an interface, not a bag of strings. A practical implementation loop looks like this:
- Inventory the input. Separate stable instructions, tools, examples, and documents from user-specific or time-sensitive data.
- Put stable content first. Move timestamps, trace IDs, live state, and the current question towards the end.
- Keep serialization deterministic. Preserve tool order, document order, whitespace where relevant, and schema formatting across requests.
- Append conversation turns. Rewriting or summarising earlier history may save tokens, but it can also change the prefix. Compare the total effect rather than worshipping the hit rate.
- Use provider controls deliberately. Automatic caching is a sensible start; explicit breakpoints and longer lifetimes are useful only when the traffic pattern justifies them.
- Version stable prompts. When instructions change, treat that as a new prefix and expect a cold cache while the new version warms.
Do not pad a tiny prompt with useless text merely to cross a cache threshold. More input can cost more, slow the request, and make the model less focused. If a genuinely useful stable prefix sits near a provider’s minimum, test the break-even point with real traffic. Otherwise, leave the inventory light.
What should you measure?
Start with the usage data your provider returns. Track total input tokens, cache-write tokens where reported, cached-read tokens, time to first token, model, prompt version, and task type. Then calculate both token hit rate and realised cost. A beautiful cache-hit percentage can still hide an expensive write pattern or a workload whose output tokens dominate the bill.
Compare a representative period before and after the prompt-layout change. Keep answer quality in the test, too. Reordering instructions for cache reuse is not a win if important constraints become weaker. The earlier guide to AI tokens and pricing explains why input, cached input, output, and reasoning categories should not be blended into one mystery number.
The short answer
Prompt caching helps when an AI application repeatedly sends a long, identical beginning followed by a smaller changing ending. Keep stable instructions, tools, and reference material first; keep live details last; and measure actual cached tokens, latency, cost, and output quality. If the prefix is short, volatile, or rarely reused, caching is another item in the inventory that weighs more than it earns. I have logged it accordingly.
