Observability and tracing
Trace every model and tool call with inputs, outputs, tokens, latency and cost so you can debug, evaluate and bill.
Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.
Key ideas
- One trace per user request; spans for retrieval, each LLM call and each tool call.
- Attach user, tenant, session, prompt version and model ID to every trace.
- Dashboards: error rate, latency percentiles, tokens and cost per route, tool failure rates.
- Collect user feedback (thumbs, edits) and link it to traces for eval datasets.
- Redact PII before traces leave your boundary.