Reference architecture: enterprise copilot

A typical copilot combines chat UI, an agent service, tools over company systems, retrieval, memory and an eval and observability layer.

Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.

Key ideas

• Edge: UI with streaming, auth (SSO), rate limiting.
• Agent service: stateless API running the agent loop; chat history in DynamoDB or Postgres keyed by session.
• Tools: MCP servers or internal functions over ERP, CRM and databases, with per-user auth.
• Retrieval: hybrid search with permission filtering.
• Platform: model gateway (routing, fallbacks, caching, budgets), tracing, eval pipeline, guardrails.
• Draw it, then walk through one request end to end including failure paths.

Go deeper

• Patterns for building LLM-based systems (Eugene Yan)