AI Agent Benchmark Integrity: What a 90% Pass Rate Can Hide

Research edition: 4 November 2026 · DN Benchmark Denominator Lab v0.1

Consider a hypothetical comparison. Two vendors advertise a 90% agent success rate. One allowed a single attempt. The other generated several candidates and used a grader to identify a passing result. A third report excludes tasks that never produced a usable outcome.

The percentages may be calculated correctly. The purchasing decision can still be wrong if those numbers are treated as interchangeable.

This is the problem of AI agent benchmark integrity: whether the reported claim retains the conditions that give it meaning. It extends DN’s acceptance-testing lab. Acceptance defines what qualifies as a result; benchmark integrity asks how the result was selected, counted and presented.

The Signal: Benchmark Operators Are Policing the Conditions Behind the Score

Terminal-Bench’s integrity update requires trajectories for passing trials and describes consequences for reward hacking and prohibited benchmark alterations. Its discussion distinguishes exploiting a task loophole from submitter changes that give an agent an advantage. Those are specific policy questions, not grounds to label every strong score suspicious.

A separate timeout clarification explains why changing time limits prevents a fair comparison. Extra execution time changes the conditions of the test. DN’s inference is that the budget belongs beside the headline, rather than hidden in a technical appendix.

SWE-bench’s evaluation guide separates dataset instances, submitted predictions, completed reports, resolved and unresolved cases, errors and likely infrastructure failures. It also describes per-instance artifacts. A run that avoids a harness crash has not necessarily resolved every task. These distinctions make raw records more useful than a single aggregate.

These sources support a practical editorial rule: preserve the evidence needed to understand a score. This article does not reproduce their leaderboards or compare current model rankings.

DN Alpha Thesis: Keep the Claim Attached to Its Evaluation Contract

DN calls the bundle of task scope, system configuration, allowed attempts, assistance, grader and reporting rule the evaluation contract. This is an analytical term, not a legal instrument or a new protocol.

Changing that contract can produce a useful new experiment. Increasing retries may be a sensible product improvement. A narrow dataset can suit a narrow buyer. Human assistance can be a valuable part of a workflow. The important step is to name the change and measure the resulting system honestly.

For a procurement decision, retain three separate questions: what happened under the test, whether the deployed system can reproduce those conditions, and whether that workload represents the buyer’s work. A score can be transparent and still be a poor fit.

Try the DN Benchmark Denominator Lab

The ledger below is entirely invented. Each task has three recorded attempts. Six tasks pass on their first attempt, another three eventually pass, and one never receives a pass. One passing attempt is labeled human-assisted and another has unknown assistance.

Change the reporting subset, attempt rule and assistance requirement. The controls recompute these recorded outcomes; they do not simulate agents or predict the effect of buying more compute.

Synthetic ledger JSONAll attempt rows CSVEight reporting views JSONReporting views CSVBlank claim cardMethodology and limits

One Ledger, Several Correct Answers

The 90% result does not establish a 90% first-attempt success rate. The 100% result is mathematically correct for its selected subset, but omitting the only no-pass task after seeing the outcomes conceals a consequential choice. A legitimate predeclared subset should be disclosed with its intended scope; the lesson is not that every subset is improper.

The 70% result also needs careful language. The tool requires the declaration none; it does not independently observe autonomy. Unknown assistance stays unknown. It is excluded from that particular numerator rather than silently being treated as absent.

Observed Any-Pass Counts Are Not a Statistical pass@k Estimate

The original HumanEval paper, Evaluating Large Language Models Trained on Code, discusses pass@k and an estimator based on multiple generated samples per problem. It also distinguishes finding a passing sample with an oracle from selecting a useful sample without that knowledge.

DN’s lab uses descriptive counts over a fixed invented ledger. It does not implement the paper’s estimator, draw samples from a model, or establish a success probability. We deliberately use “any recorded attempt” rather than presenting the toy result as a formal pass@3 measurement.

This matters operationally. If a hidden grader can identify the best candidate after inspecting three attempts, ask what will choose the candidate in production. A selector may consume additional compute, need privileged evidence, make mistakes or require human review. Evaluate that component rather than assuming it is free and correct.

Nor should a buyer transform one reported rate into a new multi-attempt forecast using an independence assumption. Errors can repeat across attempts, tasks can share dependencies, and the evaluated workload may differ from deployment.

Failures, Replacements and Missing Evidence Need Separate Records

An infrastructure outage is not automatically evidence that a model cannot solve a task. It is also not a reason to erase the operational experience. Keep the original attempt, the cause evidence, the replacement link and the cost of both.

A capability report may use the benchmark operator’s prescribed treatment of environment failures. An operations report asks whether the system delivered within the agreed time and budget. Publishing both measures, with their scope, is more useful than forcing every failure into one category.

DN’s sample retains one unresolved trial. It does not qualify as a pass, remains visible and retains its modeled cost. A task with no recorded attempts would also remain in the chosen task denominator. If a reporting subset contains no tasks, the tool returns an unavailable rate rather than zero or a perfect score.

What the Score Leaves Out of the Bill

Each synthetic attempt costs an invented $1, so all thirty recorded attempts total $30. The lab keeps that full-ledger cost visible when the reporting view changes. This is not a counterfactual estimate of what a first-attempt-only service would have cost.

The example omits infrastructure, grading, human intervention and candidate-selection costs. A commercial comparison must add the costs actually incurred by the evaluated workflow and identify whose bill they are on. Treat undisclosed amounts as missing rather than automatically zero.

For implementation, connect a benchmark claim to the real API-call cost model and outcome receipts. Keep the same cohort and time period when dividing costs by qualifying outcomes. A leaderboard metric and a customer’s accepted-outcome bill can have different denominators.

From Benchmark Claim to Purchasing Decision

1. Capture the claim: retain its URL, publication date, exact metric and stated scope.
2. Request the evaluation contract: task IDs, versions, budgets, tools, grader and assistance policy.
3. Reconcile the denominator: connect every eligible task to attempts, omissions, unresolved outcomes and replacements.
4. Inspect selection: determine whether the reported success relies on a grader or reviewer unavailable in production.
5. Run a scoped pilot: use a representative, permitted workload and define acceptance before execution.
6. Compare the full system: assess useful outcomes, latency, review burden and cost under matched conditions.

A beginner can start with the first three questions before accepting an advertisement. A practitioner can reconstruct the task-to-attempt ledger. An institution can require a versioned claim card, a scoped pilot and a designated owner for unresolved evidence.

Small score differences need an uncertainty analysis suited to the experiment. Repeated attempts on one task do not create the same evidence as equally many independent tasks. This invented example offers no useful confidence interval for an empirical population.

Methodology, Reproduction and Falsification

Evidence classes: official benchmark guidance and the original HumanEval paper support the methodological distinctions. The evaluation-contract framework is DN’s interpretation. All ten tasks, thirty results, assistance declarations and costs in the lab are invented.

Executed work: the published summarizer computes eight combinations of reporting subset, attempt rule and assistance rule. All eight reproduce from the released source and ledger. There were no model calls, provider tests, benchmark submissions or paid evaluation runs.

Reproduce: save ledger.json, results.json, summarize.mjs and reproduce.mjs in one directory. Run node reproduce.mjs, without dependencies or network calls. The result file identifies the summarizer’s SHA-256 bytes; a hash does not authenticate a publisher or prove an execution date.

Supported scope: the function accepts bounded synthetic ledgers with unique task IDs, zero through three ordered attempts, explicit result and assistance labels, and nonnegative integer USD-cent costs. It rejects malformed policies and unsupported values rather than guessing. The public browser tool only uses the fixed example; it collects no private benchmark logs.

Falsification: supply a supported ledger and policy that the source miscounts. A different answer after changing the policy is not itself a bug. Establishing contamination, undisclosed assistance or prohibited benchmark changes requires additional evidence and the operator’s rules.

Disclosure, reuse and updates: Research Desk is DN’s paid service and does not determine the results. DN-authored research files are CC BY 4.0; retain attribution and limitations. Third-party sources retain their terms. Initial release: v0.1, 4 November 2026. Material revisions require a new version. Send corrections with the policy and reproducible evidence.

Primary Sources

1. Terminal-Bench: Leaderboard Integrity Update, current page reviewed 4 November 2026.
2. Terminal-Bench: Leaderboard Integrity and Timeouts, including its 10 September 2025 update.
3. SWE-bench: Evaluation Guide, reviewed 4 November 2026.
4. Chen et al., Evaluating Large Language Models Trained on Code, v2, 14 July 2021.
5. DN v0.1 methodology, synthetic ledger and executed reporting views.

Frequently Asked Questions

What is AI agent benchmark integrity?

It is the discipline of making a reported result traceable to its tasks, system configuration, attempt budget, grader, assistance policy and exclusions. A transparent result can still have limited relevance to a particular buyer.

Does a high benchmark score mean someone cheated?

No. Strong performance, useful retries and legitimate task subsets can all produce high scores. An omission calls for clarification; a misconduct finding requires specific evidence and the applicable benchmark rules.

What does the DN lab measure?

It recomputes descriptive counts from ten invented tasks and thirty invented attempt records under eight reporting policies. It does not run models, authenticate logs or estimate production reliability.

Why do the same records produce 60%, 90% and 100%?

Six of ten tasks pass on their first recorded attempt; nine pass at least once across three recorded attempts. Omitting the only task without a pass produces nine of nine. The numerator, attempt rule and denominator must travel with the percentage.

Is this a pass@k estimator?

No. The lab counts observed qualifying tasks in a small invented ledger. Formal pass@k evaluation has a defined sampling and estimation procedure. Do not substitute a probability formula or a toy count for that procedure.

Does any successful attempt mean an agent can pick the correct result?

No. The grader can identify a passing candidate after inspecting all attempts while the deployed agent may lack a reliable selector. Evaluate selection and its cost separately.

Why is unknown assistance excluded when none is required?

Unknown does not establish that no assistance occurred. The lab preserves that label and excludes it from that particular numerator. A declaration of none is still not independent verification of autonomy.

Should infrastructure failures count as model failures?

Not automatically. Preserve their records and cause evidence. Separately report capability measures under the benchmark rules and operational completion for the intended workload; disclose reruns and replacements.

Can I infer a confidence interval from this synthetic example?

No useful empirical population inference follows from this invented sample. A real analysis must account for its sampling plan, repeated tasks, dependence, uncertainty and intended workload.

How can I reproduce or reuse the DN files?

Save ledger.json, results.json, summarize.mjs and reproduce.mjs in one folder and run node reproduce.mjs. DN-authored files are CC BY 4.0 with attribution and limitations. The blank claim card is a disclosure template, not certification.

Evidence cut-off: 4 November 2026. DN Benchmark Denominator Lab v0.1. A recomputed percentage is not an authenticated benchmark claim or a forecast of deployment reliability.