Prompt caching
Providers can cache a repeated prompt prefix so later calls skip reprocessing it, cutting input cost and latency substantially.
Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.
Key ideas
- Order prompts from most static to most dynamic: tools, system prompt, documents, history, then the new message.
- Anthropic uses explicit cache breakpoints; OpenAI and Gemini cache eligible prefixes automatically.
- Caches expire after a short TTL (minutes); longer TTLs cost more to write.
- Monitor cache read versus write tokens in usage metadata to confirm hits.
- Biggest wins: agents with large tool sets, RAG over a fixed document, multi-turn chat.
Common mistakes
- Putting a timestamp or user ID at the top of the system prompt, which breaks the cache on every call.