Evaluating each step, not just the answer

An agent can produce a perfect reply after reading account data before verifying the customer, issuing a credit nobody approved, or creating the same dispute twice. A final-answer score gives all three full marks. Evaluate agents at three levels. The outcome: the final answer and, more importantly, the final state of the systems it touched. The trajectory: which tools it called, with which arguments, in what order, how efficiently, and how it recovered from errors. The components: retriever, router, extractor, tools and guards, each tested in isolation with fixtures so failures can be traced to an owner. Get the trajectories from trace spans. Match them against a specification using the loosest mode that still captures the requirement, with hard constraints (forbidden tools, precedence) checked separately. For conversational agents, simulate users, grade the final state and report pass^k, because customers get one attempt each.

• Outcome is state, not text. "Your case is created" is a claim; the case table is the fact.
• Constrain order with precedence rules, not exact sequences, unless the workflow is genuinely fixed.
• Forbidden calls and precedence violations are hard failures regardless of the final answer.
• Component evals tell you what to fix; end-to-end evals tell you whether it matters.
• Reliability is pass^k, not pass@1.