I have a confession about my own learning process: sometimes I read a doc, nod along, and only realize weeks later that I never actually understood it.
Prompt caching was like that for me. Every provider's docs say some version of "caching saves compute and reduces cost," and I nodded along for months. Then one day I actually stopped and thought about it: wait, what does "saves computing power" even mean here? The prompt still gets sent over the network. The text is identical. Wouldn't caching just save… bandwidth?
If you had the same half-formed thought, this post is for you. Because the real answer is much more interesting — and once the mental model clicks, a bunch of 2026-era pricing decisions suddenly make sense.
The part I was missing: prefill vs. decode
Here's the thing that finally made it click. When an LLM handles your request, there are two very different phases:
Prefill — the model ingests your entire prompt. Every single input token gets run through the attention layers, and the model computes a set of intermediate values called KV tensors (key-value pairs, one set per token, per layer). For a long system prompt, this is the expensive part. Thousands of tokens × dozens of layers = a mountain of matrix math.
Decode — the model generates its response, one token at a time, using those KV tensors as its "memory" of what you asked.
Now the key insight: if you send the same prefix twice — same system prompt, same tool definitions, same documents — the KV tensors for that prefix are identical every time. Recomputing them is pure waste. It's like re-planning an entire road trip from your driveway every time you stop for gas, instead of just… continuing the trip.
Prompt caching is the provider saying: "We'll keep those computed tensors on our GPUs. If your next request starts with the exact same prefix, we skip the prefill for it and only process what's new."
So no — it's not about bandwidth at all. Your full prompt still travels over the wire, byte for byte, on every request. What's saved is the FLOPs: the redundant attention computation that would otherwise happen again and again. And since you're billed per token processed, skipped computation shows up as a direct discount.
Why this went from "nice feature" to "essential" in 2026
Two words: agent loops.
Think about what an AI agent actually does. Turn 1: it gets your system prompt, tool definitions, and your message. Turn 2: it gets the system prompt, tool definitions, your message, and its first response, and a tool result. Turn 3: all of that, plus more. A 50-step agent run re-sends the same system prompt and tool schemas fifty times. Chat apps do the same thing — every turn replays the whole conversation history.
Without caching, you pay full input price for all of it, every time. With caching, everything up to the newest bit reads from cache at a steep discount. This is why the providers went hard on caching this year — agentic workloads made repetition the dominant cost on the entire platform.
And the numbers got aggressive. As of mid-2026, all three major providers have converged on roughly 90% off cached input tokens:
- Anthropic gives you explicit control: mark blocks with
cache_control, cache reads bill at 10% of input price, writes cost 1.25× (5-minute TTL) or 2× (1-hour TTL), up to 4 cache breakpoints per request. The write surcharge breaks even after about two reads. - OpenAI went fully automatic — no code changes, no write premium. Anything past a 1,024-token prefix is eligible, and on the GPT-5.x family cached input is 90% off. Newer models even retain the cache up to 24 hours.
- Google does both: implicit automatic caching on Gemini 2.5+, plus an explicit API where you pay storage-by-the-hour for a named cache you control.
One warning from my research: a lot of older posts still quote 50% for OpenAI and 75% for Gemini. Those figures are stale — that was the 2024–25 era. The market moved.
The gotchas that will absolutely get you
The discount is real, but the rules are strict:
Exact prefix match or nothing. Caching matches from the start of your prompt, token for token. One stowaway timestamp, request ID, or "Hello, Tun!" greeting in your system prompt and your hit rate is 0%. I cannot overstate how many "caching doesn't work" complaints trace back to a dynamic string at position one.
Static first, dynamic last. The correct order is: tool definitions → system prompt → documents → few-shot examples → conversation history → the new user message. Everything stable goes up front so the cacheable prefix is as long as possible.
Minimums exist. OpenAI won't cache under 1,024 tokens. Anthropic's floor varies by model (1,024–4,096). Below that, you're paying full price no matter what.
Output is never discounted. Every token the model generates goes through the full pipeline, every time. Caching only touches the input side. And there's zero quality trade-off — the cached KV tensors are mathematically identical to fresh ones, so responses don't change.
So why do subscriptions and APIs behave differently?
This was the second half of my confusion, and the answer is simpler than I expected: caching is a property of the request path, not the model.
Whether you actually benefit depends on how the call is constructed — whether the client marks cacheable blocks, whether the model tier supports it, how prompts are assembled, even which server your request lands on. A chat app subscription and a raw API call are two different plumbing systems delivering the same water. The model is the same; the route is not.
Which leads to a mildly uncomfortable conclusion if you build with these APIs: your prompt structure is your cost architecture. Where you put your timestamps, how you order your blocks, how your conversation history grows — those are billing decisions now, not style choices.
The one-line version
Prompt caching doesn't save what you send. It saves what the model would have to rethink.
Once I internalized that, I started seeing my own systems differently: every repeated token is either a wasted dollar or a cached one, and the difference is mostly a few structural choices. If you run anything agentic, go audit your system prompt for stowaway dynamic strings right now — that ten-minute check is the cheapest optimization you'll do all year.
