Agent evaluation is undergoing a paradigm shift from 'it works' to 'safe and reliable'. The latest agent evaluation suites cover four key dimensions. First, Multi-turn Safety — evaluating whether the agent maintains safe behavior boundaries across multiple conversation turns, especially under adversarial user pressure. Second, Tool Call Hallucination detection — when a tool returns empty or error results, does the agent report truthfully or fabricate data? Third, Permission Abuse Tendency — agents with file read/write, network access, etc., whether they overuse permissions in certain scenarios. Fourth, combined Task Completion Rate and Efficiency evaluation — not just whether the task is done, but whether token consumption and step count are reasonable.
In the Zero to Pro Agent Harness course, these evaluation dimensions directly correspond to pre-deployment acceptance checklists. Learners should understand: agent evaluation is not a one-time 'exam' but should be embedded in CI/CD as a continuous safety guardrail. The course recommends: define an eval spec for each Agent Skill or MCP tool, covering three tiers — functional tests, safety tests, and robustness tests. Only when an agent passes all three tiers is it allowed into production. The new eval suites from Anthropic and OpenAI provide reusable eval templates, significantly lowering the startup cost of safety evaluation.