Cost and latency optimisation

Most savings come from sending fewer tokens to smaller models, and from caching.

Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.

Key ideas

• Model routing: classify difficulty and send easy requests to a small model.
• Prompt caching for repeated prefixes; semantic caching for repeated questions.
• Batch APIs for offline jobs at around half price.
• Streaming improves perceived latency; show partial output and tool progress.
• Trim context: shorter system prompts, fewer tools, compact tool outputs.
• Build a cost model: requests per day x (input + output tokens) x price, per route, with cache hit rates.

Common mistakes

• Optimising cost before quality is proven.