Agent evals: suites, rubrics, and judges that lie

A test asks "did the code do what I specified?" An eval asks "did the agent accomplish the task?" — over a distribution, statistically, with no single correct output. Three pieces: a task suite (the distribution), a rubric (what counts as success), and a grader (who decides). The grader is where nearly everything goes wrong.

Graders come in three kinds, in descending order of trustworthiness:

1. Execution-based — run the tests, diff the database, check the file. Objective, cheap to trust, hard to author.
2. Reference-based — compare against a known-good answer with string or semantic matching. Middling.
3. LLM-as-judge — ask a model. Flexible, scalable, and biased in ways that are measured, reproducible, and larger than the effects you are trying to detect.

Run a task n times, succeed c times. Two metrics, and the difference between them is the difference between a demo and a product.

pass@k = 1 - E_task[ C(n-c, k) / C(n, k) ]
pass^k = E_task[ C(c, k) / C(n, k) ]

pass@k is the probability that at least one of k attempts succeeds. pass^k, in τ-bench's own words, is "the chance that all k i.i.d. task trials are successful, averaged across tasks."

They diverge violently. GPT-4o on τ-bench scores pass^1 of 61.2% on retail and 35.2% on airline (48.2% average). By pass^8 on retail, that 61.2% has collapsed to roughly 25%.

Which one you report is determined by one property of your task: reversibility.

• A code-generation task where you compile candidates and keep whichever passes — pass@k is honest. Failed attempts cost compute and nothing else.
• An agent that issues refunds, sends emails, deletes files, or moves money — every attempt lands in the world. There is no "keep the best of 8."

τ-bench's framing is worth memorizing: reporting pass@k for an agent taking irreversible actions is a category error. It is not a slightly optimistic metric; it measures a quantity that does not exist for that system. And note what pass^k really surfaces — consistency. A 61% agent that is 61% on every task is a different product from one that is 100% on 61% of tasks and 0% on the rest, and pass^1 cannot tell them apart.

These are measured effects from MT-Bench (arXiv:2306.05685, NeurIPS 2023 Datasets & Benchmarks). Treat the numbers as calibration, not trivia.

Position bias. Swap the order of two answers and see if the verdict holds. On near-identical answers, consistency was: Claude-v1 23.8%, GPT-3.5 46.2%, GPT-4 65.0%. Claude-v1 showed a 75.0% first-position bias, and part of it was name bias — renaming the assistants lifted consistency to 56.2%. Few-shot prompting pushed GPT-4 to 77.5% at roughly 4× the API cost.

A separate result makes the stakes concrete: "LLMs are not Fair Evaluators" (arXiv:2305.17926) found that reordering alone made Vicuna-13B beat ChatGPT on 66 of 80 queries (82.5%). Reordering. Nothing else changed. Its proposed mitigations — Multiple Evidence Calibration, Balanced Position Calibration, Human-in-the-Loop Calibration — are all forms of making the judge vote more than once under varied conditions.

Consistency is also not accuracy. A judge that is 100% consistent and 100% wrong scores perfectly on this test.

Verbosity bias. Take 23 list-format answers and pad them from 5 items to 10, adding no new information. Fooled: Claude-v1 91.3%, GPT-3.5 91.3%, GPT-4 8.7%. The detail that should worry you: all judges correctly tie identical answers. The naive bias test passes cleanly while the real failure hides one step away.

Self-preference. Judges favor their own output. MT-Bench measured roughly +10 points for GPT-4 and +25 for Claude-v1, but its authors explicitly state they "cannot determine whether the models exhibit a self-enhancement bias" — the confound is that a model may simply be right about its own quality. The causal result comes from arXiv:2404.13076 (Panickssery, Bowman, Feng, NeurIPS 2024), which established self-preference causally by fine-tuning models to vary self-recognition and observing self-preference move with it. Cite that one.

Reference-guided grading is the mitigation that actually works. On math and reasoning, GPT-4 judged an incorrect answer correct 14/20 times with a default prompt, 6/20 with chain-of-thought, and 3/20 when given a reference answer. Chain-of-thought halves the error; a reference answer nearly eliminates it. If you can produce a reference, produce one — it beats prompt engineering by a wide margin.

Judges are not useless. GPT-4 agreed with human experts 85% of the time on MT-Bench (S2, excluding ties), against 81% human–human agreement, and agreement rises toward 100% as the capability gap between compared answers widens. In the disagreements, humans judged GPT-4's reasoning reasonable 75% of the time and changed their own choice 34% of the time. Judges are good at easy calls and unreliable at close ones — which is exactly backwards from what you need, because close calls are where your regressions live.

The survey literature (arXiv:2411.15594) splits these into task-agnostic biases (diversity, cultural, self-enhancement) and judgment-specific ones (position, style, verbosity, authority). Useful taxonomy: the first group you mitigate by changing the judge, the second by changing the protocol.

Grading "did the agent correctly refactor this function?"

Bad: ask a judge "is this good?" — verbosity bias rewards the longer diff, position bias decides your A/B test, and self-preference tilts toward whichever model wrote the candidate.

Better, in layers:

1. Execution first. Does it compile? Do existing tests pass? Both are objective and answer most cases without a judge at all.
2. Structural checks. Did untouched files stay byte-identical? (Straight from the e2e rule — and it catches the agent that "refactored" by deleting the tests.)
3. Judge last, reference-guided, on what remains: given this reference refactoring, is the candidate equivalent in behavior? Score against the reference, not in the abstract.
4. Randomize position across every pairwise call and report consistency alongside the score. A run with 60% positional consistency has not measured anything.

The ordering is the lesson: every question you can answer by execution is a question the judge cannot get wrong.

Execution-based grading is objectively better and dramatically more expensive to author — you need a runnable environment and real verification tests per task, which is why Terminal-Bench ships a Docker image per task and why SWE-bench is confined to repositories that already had test suites.

And execution-based grading is not automatically correct. SWE-bench's own history is the cautionary tale, and it is worth carrying in full because every one of these defects can appear in a suite you write:

• Solution leakage. SWE-Bench+ (arXiv:2410.06992) found 32.67% of instances had the solution present in the issue text or comments. The agent was reading the answer.
• Weak tests. A further 31.08% passed via tests too weak to distinguish a real fix from a plausible one.
• The combined effect: SWE-Agent + GPT-4 dropped from 12.47% to 3.97% after filtering — roughly 3.1× inflation. Over 94% of issues predate model training cutoffs, and the defects persist in both Lite and Verified.
• Plausible-but-wrong patches. arXiv:2503.15223 (ICSE 2026) ran differential testing on SWE-bench Verified and found 29.6% of plausible patches behave differently from ground truth, 28.6% of those certainly incorrect — inflating reported rates by 6.2 absolute percentage points.
• Construct validity. Epoch AI's analysis: 91% of tasks take an experienced engineer under an hour, 39% are trivial (<15 minutes, ~5 lines changed), only 3 issues exceed 4 hours. Django alone is ~50% of issues and the top 5 repositories are >80%. Half predate 2020. 87% are bug fixes, 9% features, 4% refactorings.

SWE-bench Verified (OpenAI, August 2024) exists because of this: 93 developers annotated 1,699 samples, finding 38.3% of problem statements underspecified and 61.1% of unit tests unfair; 68.3% of instances were filtered out, leaving 500. Two consequences follow, and both are commonly botched. Verified is a correction, not a difficulty-tuned subset — roughly two-thirds of the original was judged defective. And comparing a Verified score to an original SWE-bench score is invalid; they measure different populations.

None of this makes SWE-bench worthless. It makes a SWE-bench number worthless without its provenance — which scaffold, which subset, which filtering.

The general form of this defect is worth naming, because it appears far outside benchmarks. A construction artifact is any regularity in how an evaluation was authored that correlates with the correct answer independently of the competence being measured. Solution leakage is one instance. So is a multiple-choice exam whose author writes each key as a fully-qualified mechanism sentence and each distractor as a terse misconception: length now tracks correctness, and a candidate who always picks the longest option scores well while knowing nothing. Both cases have the same shape — the score is achievable by exploiting how the test was built rather than by having the ability it claims to measure. This is the module's through-line arriving at its sharpest: the number measures the harness, not the subject. When you author an eval, adversarially audit it with a knowledge-free strategy — pick the longest, pick the first, pick the one mentioning the most identifiers — and if any of them clears your pass threshold, your instrument is broken regardless of how sensible each individual item looks.

dsh does not ship a scored eval suite, and that is itself a position worth naming: its investment is in the assembled-behavior tier (M8.2) and in per-file coverage, on the argument that a harness's own regressions are deterministic and should be caught deterministically. The FAIL_TO_PASS/PASS_TO_PASS structure appears in a different form — packages/core/agent-loop/tests/contract-regressions.spec.ts holds permanent tests for contract regressions, each one a task that must keep passing forever.

Where dsh does align with eval practice is the anti-cheating stance: assert the world, never the self-report.

τ-bench puts an LM in the user's seat and grades by final database state — no judge, no rubric text, just a diff. Its scale is modest and deliberate: τ-retail is 115 tasks over 500 users, 50 products, 1,000 orders, with 7 write and 8 non-write tools; τ-airline is 50 tasks over 500 users, 300 flights, 2,000 reservations, 6 write and 7 non-write tools.

τ²-bench (arXiv:2506.07982) extends this to dual control: agent and user both act on a shared environment, formalized as a Dec-POMDP, in a Telecom domain with a compositional task generator and a constrained user simulator. Its headline finding is a rebuke to the usual reasoning-benchmark framing — communication and user-guidance quality dominate, not reasoning ability. Its ancestry also carries the SWE-bench lesson forward: tau3-bench (2026) adds voice and knowledge retrieval and ships 75+ task fixes. τ-bench had its own data-quality debt. Every benchmark does.