An AI agent says it moved a support ticket to the right team. The message sounds confident. The ticket is still sitting in the old queue. That fictional example captures the difference between an agent explaining a job and actually finishing it.
Check the outcome outside the agent’s own message, then check the evaluator too. In our hypothetical 100-trial example, a grader agrees with the human reference 86% of the time but catches only half the failures. Overall agreement and failure detection answer different questions.1, 2
One hypothetical test, two very different impressions
86 of 100 grader decisions match the human reference in this invented example.
IllustrativeOnly 10 of the 20 reference failures are caught by the same grader.
IllustrativeAnthropic’s agent-evaluation guidance distinguishes the conversation record from the outcome in the environment. A booking confirmation in the chat is not proof that the reservation exists. The test needs access to the evidence that would settle the question.1
Ask for a receipt, not another confident answer
For a support-ticket task, inspect the assigned queue. For a calendar task, check the actual event and attendees. For research, open the cited source and see whether it supports the claim. The right witness depends on the job, but it should not be only the agent’s own summary.
Some tasks also need judgment: was the reply helpful, was the explanation fair, did it omit something important? An AI evaluator can help with that review. It is still another system that needs checking, not an independent guarantee because it was given the job title “reviewer.”
Why 86% can be less reassuring than it sounds
Suppose a person reviews 100 trials and identifies 80 acceptable results and 20 failures. An AI grader then judges the same trials. The following counts are invented to show the calculation, not results from an agent or model we tested.
| AI grader says | Human: acceptable | Human: failed | Total |
|---|---|---|---|
| Pass | 76 | 10 | 86 |
| Fail | 4 | 10 | 14 |
| Total | 80 | 20 | 100 |
The grader agrees on 76 acceptable results and 10 failures: 86 correct matches out of 100. But it passes the other 10 failures. Its failure-detection rate is therefore 10 out of 20, or 50%. That matters far more when a missed failure sends a wrong answer to a customer.
Ten of the 86 results marked “pass” are wrong against the reference. Four acceptable results are rejected. Those are different mistakes with different costs. A good test report shows both, rather than turning them into one reassuring average.
Make the evaluator work for its score
Include cases where the agent sounds convincing but misses the task. Also include valid results phrased awkwardly, incomplete records, and actions that should not have been taken. Label the cases consistently before using them to assess the grader. The reference itself needs review when reasonable people disagree.
The 2023 MT-Bench study examined issues including position and verbosity bias in model-based judging. Those findings concern the models and conditions tested then. They are reasons to check your current evaluator, not a measurement of a 2026 model’s error rate.2
Questions a useful evaluation should answer
- Did the required action happen, with evidence outside the agent’s final message?
- How often did the evaluator pass a result that should have failed?
- How often did it reject useful work and create unnecessary human review?
- Does the outcome change when presentation order or answer length changes?
A score should lead to a narrower decision
A first trial does not have to justify handing over the whole workflow. It may justify letting the agent draft replies but not send them, update low-risk fields but not approve payments, or handle a narrow category before expanding. Define the decision before choosing a pass threshold.
Compare the candidates on the same work, and show the unresolved cases. A system that declines hard tasks can look excellent on the tasks it accepts. That can be a sensible design, but the human work it leaves behind belongs in the cost calculation.
The most useful question is not “How good is the score?” It is “Which mistakes can still get through, and what happens when they do?” That is the difference between a promising demo and a reason to trust the next action.
Sources and methodology
Sources checked September 21, 2026. Dates and periods for individual figures are stated beside them.
- Anthropic: Demystifying evals for AI agents ↗Accessed 2026-09-21
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena ↗Accessed 2026-09-21
Scope and assumptions
All trial counts and calculated rates are illustrative. No AI model or product was tested for this article.
The MT-Bench findings describe a 2023 study, not current model performance. Human reference judgments can themselves require review.
AI-assisted research and editing. Our editorial standards.