Evaluating agents
Agents need evals of the final outcome and of the path: tool choice, arguments, number of steps and cost.
Changes fast: check the latest updates too. Reviewed 17 Oct 2026.
Key ideas
- Outcome: did the task succeed? Check end state (DB row created, file correct) rather than text.
- Trajectory: correct tools called, valid arguments, no unnecessary or unsafe calls.
- Efficiency: turns, tokens, latency, cost per task.
- Run each task several times; agents are non-deterministic, so report pass@k or success rate.
- Use sandboxed or mocked tools so evals are repeatable and safe.
Common mistakes
- Grading exact trajectories too strictly; there are often several valid paths.