agent evaluation

Agents fail in the middle, not just the end, so final-answer-only scoring hides the real problems and rewards lucky paths. The signal is evaluating the whole trajectory and localizing where it broke. Here is the answer.

Final-answer accuracy hides how an agent got there. The signal is trajectory-level metrics: tool-selection and argument accuracy, step efficiency, and matching against reference paths, plus when exact-match is the wrong yardstick.

Single-turn evals cannot catch failures that history causes, because they have no history. The offline answer is two suites, one deterministic and one simulated, and an honest account of why the simulated user will lie to you about how good your agent is.