How to Read an AI Agent Eval Score
A trustworthy AI agent eval score comes from a sealed, independent task bank and ships with a confidence interval — self-reported vendor metrics, run on tasks the vendor chose, are marketing until a third party verifies them.
An eval score is supposed to tell you how good an AI agent actually is — but a number is only as trustworthy as how it was produced. This post explains how to read one: what a sealed task bank is and why it matters, what pass criteria mean, why every score needs a confidence interval, and why self-reported vendor metrics are not enough. It contains no specific scores, because real scores come from running the eval — not from a blog post.
Sealed task banks
A credible eval runs the agent against a versioned task bank that is kept private and rotated — never published. The reason is contamination: if the test questions are public, a high score can reflect memorization or training on the test rather than genuine capability. A sealed, rotating bank means a high score reflects the agent doing the work on tasks it has not seen. When you read a score, the first question is: was it measured on a sealed bank, or on tasks the vendor hand-picked?
Pass criteria, held out from the seller
A score needs a definition of "correct." Good evals grade outcomes against a held-out rubric — pass criteria defined independently of the seller, so the seller cannot grade its own homework. For a support agent that might be whether the ticket was actually resolved; for a coding agent, whether the change passed the tests. The grading standard should be fixed before the agent runs, not adjusted afterward to flatter the result.
Confidence intervals
Every honest eval score ships with a confidence interval and the sample size behind it. A score of 84 with a tight interval is a different claim than 84 with a wide one — the second means the eval simply has not seen enough tasks to be sure. A single bare number with no interval is a red flag: it is hiding how much uncertainty is in the measurement. Always read the interval, not just the point estimate.
Why self-reported metrics are not enough
The most common eval number you will see is the one the vendor reports about itself — on tasks the vendor chose, graded by criteria the vendor set. That is not necessarily dishonest, but it is unverifiable, and unverifiable quality claims are the exact problem that sank the first wave of AI agent companies: buyers had no neutral way to check what they were buying. A third-party score on a sealed bank is the difference between "trust us" and "verify it."
Reading a score in context
A quality score is one half of a buying decision; price is the other. An agent with a high eval score at a high price may be worse value than a slightly lower-scoring agent at a much lower price. Read the eval score (quality) next to the per-outcome price benchmarked against the WLI (cost), and compare quality-per-dollar — not either number alone.
Methodology note
This article explains how to read an eval score and deliberately cites no specific score, because real scores come only from running a sealed eval. WorkForce’s free eval produces a verified score with its confidence interval; see /methodology for how scoring and grading are governed.
Next steps
Get a free, independent eval score for any agent at /eval, and read it alongside the live per-task market rates at /wli.