Prompt caching: a narrower scope paid off more — 12 conditions measured

1. Executive summary

A token is the basic unit in which a model processes text and in which you are billed. But comparing per-token prices alone does not explain what an agent costs. Agent cost depends on how many times the same content is sent again, and the feature that reduces this repeated sending is prompt caching. When consecutive requests begin with identical content, that shared part is reused instead of being recomputed.

2. How one question becomes dozens of charges

In an ordinary chatbot, one question is one call. Agents are different. They call a tool, read the result, decide again and call again. Each of these iterations is a separate API call, and each one is billed.

Cost is the number of calls × the tokens resent per call. Count only the calls and you miss the second factor — because every call resends the entire context so far. The system prompt, tool definitions and the conversation so far are counted again as input tokens on every call. In their experiments, the paper's authors used a 10,000-token system prompt and varied the number of tool calls from 3 to 50. At 50 calls, the same 10,000 tokens are sent 50 times.

3. How much does caching actually save?

Lumer et al. (arXiv:2601.06007) compared three caching strategies across three providers (OpenAI, Anthropic, Google). On a benchmark in which agents make real web search tool calls over many rounds of requests and responses (DeepResearch Bench), they ran more than 500 agent sessions and measured API cost and time to first token together.

The paper compared these three strategies.

In its abstract, the paper sums up the results as 41–80% lower cost and 13–31% shorter time to first token. But open the per-model table and the range is wider, and which strategy did best also differs by model. Here are all 12 conditions as reported.

Source · checked against the table in arXiv:2601.06007. For time to first token, a negative value means shorter and a positive value means longer latency. Bold marks the best value in that column for each model.

4. Where caching broadly backfires

Cost went down in all 12 conditions. But time to first token actually got longer in two conditions: 8.8% for GPT-4o with full-context caching, and 2.9% for Gemini 2.5 Pro with tool results excluded. The paper's explanation is that “caching dynamic content can introduce overhead that negates latency benefits.”

The three rules the paper sets out all come down to one principle: keep content that changes with every request out of the cached portion. The third rule, however, depends on the model, as the table above shows.

5. Caching terms differ by provider

Under the same name, “prompt caching,” the way you turn it on, the minimum length and the retention time all differ. The values below were checked directly in each provider's official documentation.

Source · each provider's official documentation (checked 2026-09-08). “Not stated” means we could not find the value in that official documentation, not that no such charge or condition exists.

6. What comes after caching

Prompt caching cuts repeated computation for requests that need an immediate response. Work that does not need results right away is better sent down a different path altogether.

7. What you can do today

8. What this is not yet

• These are not our own measurements. Every reduction figure in this article comes from arXiv:2601.06007; we did not measure them again. With a different workload, the numbers will differ.
• The abstract's range does not capture the whole table. The abstract gives 41–80% for cost and 13–31% for first token, but the actual range across the 12 conditions is 27.8–81.4% for cost and −30.9% to +8.8% for first token. We could not confirm from the paper which subset the abstract's figures summarize. The body of this article is based on the table values.
• The model lineup is as of the measurement. The paper's models are GPT-5.2, GPT-4o, Claude Sonnet 4.5 and Gemini 2.5 Pro. Caching behavior may have changed in models released since.
• The benchmark is a web search agent. DeepResearch Bench is a task that calls a search tool repeatedly. This paper does not answer whether the results carry over to work where context builds up differently, such as code editing.
• Prices are as of the date checked. The table values were read from each provider's documentation on 2026-09-08. Recheck them before you adopt.
• We could not confirm Google's savings ratio. The official documentation says only that savings are applied automatically when the cache is reused, and gives no specific ratio. We left it blank.

9. Sources

We checked every figure in this article directly in the five sources above. We did not use figures from secondary roundup blogs, because we could not find their original sources. Prices and cache behavior change often, so recheck the official documentation before you adopt them.

Revised 2026-09-21. At publication, this article stated the cache-read price, 0.1× the base input price, as a rule that holds for every model. Rechecking the official pricing page on 2026-09-21 showed that Claude Fable 5.1 and Mythos 5.1 are at 0.025× ($0.25 per million tokens), a fourfold difference from the other models. We added the exception in three places: the summary KPI, the provider comparison table and the sources table. The measurements and conclusions are unchanged — the judgment that caching a narrower portion is better comes from how often the cache is invalidated, not from the multiplier. How the multipliers that now differ by model, and the tokenizer change, affect price comparisons is covered separately in You cannot compare model costs from the price list alone.