Reliability patterns

LLM APIs fail, throttle and slow down. Build the same resilience you would for any external dependency.

Stable: this rarely changes. Reviewed 17 Oct 2026.

Key ideas

• Retries with exponential backoff and jitter on 429 and 5xx; respect retry-after headers.
• Fallback models or regions; cross-region inference on Bedrock.
• Timeouts per call and per request; stream to detect stalls early.
• Idempotency keys for tool actions with side effects.
• Queue long jobs and return a job ID instead of holding HTTP connections.