Prompt Caching

What Is Prompt Caching?

Prompt caching is an inference optimization that stores the computed internal state of a repeated prompt prefix, so later API calls reusing that prefix skip reprocessing it. For agents that resend a large system prompt, tool definitions, and growing conversation history on every turn, it cuts input cost dramatically and reduces time to first token.

Key Takeaways

• Caching works on exact prefixes. Everything before the first changed token can be reused; everything after it is recomputed, so stable content must come first.
• The savings are large and real: cached input tokens are typically billed at a small fraction of the normal rate, often around a tenth, and long prompts start responding noticeably faster.
• Agent workloads are the ideal case, because each turn resends the same system prompt, tool schemas, and prior transcript with only a little new material appended.
• Cache entries expire after minutes of inactivity on most providers, so the economics favor active sessions and high-traffic prompts, not occasional calls.

How It Works

Before generating anything, a transformer model processes the entire input prompt and builds up internal attention state, stored in what is called the KV cache. That prefill work scales with prompt length and is a major share of inference cost. Prompt caching keeps that computed state on the provider's servers after a request finishes. When the next request arrives with a byte-identical prefix, the server loads the stored state and begins processing at the first new token instead of at token zero. The gains scale with prefix length: Anthropic reports savings of up to 90% on cost and 85% on latency for long prompts, and in its 2025 published example, chatting with a cached 100,000-token document cut time to first token from 11.5 seconds to 2.4 seconds [1].

The exact-prefix rule drives all the practical engineering. Providers differ in mechanics: some require explicit cache breakpoints marked in the request, others cache automatically above a minimum prefix length, and cache lifetimes range from a few minutes by default to longer windows at extra cost. Azure OpenAI, for example, documents a minimum identical prefix of 1,024 tokens before caching kicks in, with cache reads discounted on input-token pricing up to a 100% discount on Provisioned deployments (2026) [2]. In every case, one changed character invalidates everything after it. Prompt structure therefore follows a stability gradient: system prompt and tool definitions first, then few-shot examples and reference documents, then conversation history, with the volatile per-request material last. A classic self-inflicted failure is interpolating a timestamp or request ID near the top of the system prompt, which silently disables caching for the whole prompt on every call.

Multi-turn agent loops compound the benefit. The transcript grows by appending, so turn N's prompt is a prefix of turn N+1's. Each turn pays full price only for the newest tool results and messages while everything earlier hits the cache. This is a large part of why long agentic coding sessions with six-figure token contexts are economically viable at all.

Example

A team runs a code-review agent on every pull request. Each run sends a 30,000-token prompt: coding standards, tool definitions, and repository conventions, followed by the diff, which averages 2,000 tokens. Without caching, every review pays for 32,000 input tokens of prefill. They restructure the prompt so the stable 30,000 tokens form the prefix and mark it as cacheable, and they keep runs warm during business hours by batching reviews. Now a typical review pays full price on roughly 2,000 new tokens plus the discounted cached rate on the rest, and reviews start streaming back sooner. Their one bug along the way: a "generated at" timestamp sat inside the standards header, invalidating the cache on every single run until someone moved it to the end of the prompt.

What People Get Wrong

The recurring misunderstanding is expecting caching to work like a web cache with fuzzy or partial matching. It does not match on similarity, and it does not cache responses. Two prompts that are 99 percent identical but differ at token 50 share only the first 49 tokens of cached work. Nothing about the model's output is stored or replayed; the same cached prefix can still produce different completions. Prompt caching saves recomputation of input, and only for prefixes that match exactly.

FAQ

Does prompt caching change the model's output? No. The cached state is mathematically identical to what recomputing the prefix would produce, so response quality and variability are unaffected. Only cost and latency change.

Is prompt caching the same as the KV cache? They are related layers. The KV cache is the in-memory attention state every transformer builds while processing any request. Prompt caching is the product feature that persists that state across separate API calls and prices reused tokens at a discount.

Is cached prompt data a privacy risk? Cached state lives server-side with the provider and is scoped so that other customers cannot hit your cache entries; major providers key caches per organization. It follows the provider's normal data retention policies, but teams with strict data-residency requirements should verify how long cached state persists and where.

Sources

1. Anthropic. "Prompt caching cost and latency reductions, including the 100,000-token document example." https://claude.com/blog/prompt-caching. Accessed August 2026.
2. Microsoft Learn. "Azure OpenAI prompt caching minimum prefix and cache-read discounts." https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/prompt-caching. Accessed August 2026.