The demonstration worked. The next run needed a correction. The third produced a different answer. Which performance did you buy?
An AI agent that succeeds once has demonstrated a capability under one set of conditions. A business deploying that agent needs evidence about the outcomes it can expect repeatedly, the exceptions it must service and the failures its controls must contain.
DN’s thesis: repeatability is an economic property of the whole workflow. Measure accepted outcomes across repeated trials, preserve the failures, and disclose the human assistance that changes the result.
What Matters
- One successful run is limited evidence. Keep every scheduled trial, including timeouts and abandoned attempts.
- Define the outcome before testing. A confident completion message does not verify a financial action.
- Separate assisted and unassisted results. Planned approvals and unplanned rescue should be identifiable.
- Use matched conditions. Record model, harness, tools, data, permissions and initial state.
- Inspect the failure distribution. A good average can hide a task category that repeatedly fails.
- Report what was tested. Repetition alone does not establish security, robustness or suitability for live money.
Evidence Block
Source review: 13 October 2026. This is a proposed evaluation framework and a reader-input analysis tool. DN has not completed the vendor comparison described here. Sample runs are fictional.
Anthropic’s evaluation guidance distinguishes finding at least one successful solution across attempts, commonly expressed as pass@k, from succeeding across all repeated trials, expressed as pass^k. These answer different product questions.[1]
Towards a Science of AI Agent Reliability proposes evaluating reliability across consistency, robustness, predictability and safety. This supports treating reliability as several dimensions rather than a single completion score.[2]
ReliabilityBench describes an evaluation approach covering repeated execution, task perturbations and controlled tool failures. Its design is relevant to this framework; DN does not treat the paper as evidence of performance in financial deployment.[3]
Evidence boundary: the sources establish evaluation concepts. They do not establish which agent is best for your workflow or provide results for DN’s proposed finance scenarios.
The Difference Between a Demo and a Service
A demonstration answers whether a system can produce a useful result. A service must also answer how often it delivers, under which conditions, at what cost and with whose help.
Imagine an agent asked to reconcile a fixed set of invoices against payment records. Its first output is correct. On later executions it sometimes misses a duplicate identifier, applies a different rounding convention or treats an unresolved payment as settled.
Those are illustrative possibilities, not measured failure rates. They explain why the original success is insufficient for a deployment decision. Repetition reveals the range of outcomes that a polished demonstration can conceal.
The same logic applies to market research. Repeated briefs need not use identical wording. They should satisfy the same evidential standard: correct figures, consistent cut-off times, traceable sources and clear separation of facts from inference.
Repeatability means meeting the requirements consistently. It does not require identical prose or identical internal steps.
Four Questions That Must Stay Separate
| Question | Test design | What it cannot establish alone |
|---|---|---|
| Can it succeed? | Attempt a specified task and verify the outcome. | Consistency across future runs. |
| Can it repeat that success? | Repeat matched tasks from documented starting states. | Performance under new task distributions. |
| Can it handle changed conditions? | Vary wording, inputs or controlled tool faults. | Resistance to every adversarial action. |
| Can it act within authority? | Test boundaries, prohibited actions and required escalation. | Complete security or legal compliance. |
Report these dimensions separately. A combined score can obscure a serious weakness when good performance elsewhere compensates for it.
Why “It Eventually Worked” Can Mislead
For illustration, assume independent trials with the same 90% success probability. The probability of at least one success in ten trials is 1 − 0.110, almost 100%. The probability of all ten succeeding is 0.910, approximately 34.9%.
These are hypothetical probabilities under strong assumptions. They are not observed results, and ten trials could share failures caused by the same outage, stale data or permission problem.
The example shows why selecting the best attempt and demanding consistent delivery produce different pictures. It does not mean that a 90% per-run success rate causes 65.1% of tasks to fail. That latter number refers to batches containing at least one failure under the example’s assumptions.
A retry can be useful, but it spends resources and may repeat a side effect. Before retrying a payment or order, resolve whether the earlier action occurred. Repeating a financial instruction is not equivalent to regenerating a paragraph.
The DN Repeatability Protocol
- Specify acceptance. State the required output, deadline, evidence, numerical tolerances and authority boundaries.
- Freeze a baseline. Record the model identifier, harness version, prompts, tools, permissions and data snapshot.
- Choose representative tasks. Include normal cases and clearly labelled exceptions. Describe how the sample was selected.
- Reset deliberately. Use the same intended starting state for matched trials. Run persistent-memory tests as a separate condition.
- Schedule trials before seeing results. Keep successes, failures, timeouts and aborts. Define retries within each trial consistently.
- Verify independently. Check the resulting records or artefacts against the rubric. Record disputed or unverified outcomes separately.
- Log human work. Distinguish routine approval, clarification, correction and manual completion.
- Compare like with like. Use the same trial count and task mix when comparing configurations. Publish exclusions and missing data.
One hundred repetitions of one easy task do not provide the same coverage as repetitions across many task types. Trial count and task coverage are separate design decisions. A pilot can reveal failure modes; sample size for a stronger statistical claim depends on the target rate, desired precision and dependence between observations.
Four Finance Scenarios Worth Testing
| Scenario | Acceptance evidence | Boundary |
|---|---|---|
| Invoice reconciliation | Known unmatched items, duplicates and totals identified within specified tolerances. | Use synthetic records or appropriately protected data. |
| Evidence-backed market brief | Figures match the frozen source pack; dates and unsupported inference are handled correctly. | Score research quality separately from trading profitability. |
| Spending-limit enforcement | Permitted simulated actions proceed; prohibited actions are blocked or escalated as specified. | Use a mock wallet or sandbox, not live funds. |
| Ambiguous payment response | The agent reconciles authoritative mock status before a permitted next action. | Predefine pending, completed and failed states and prevent real transfers. |
These are proposed test cases, not tasks DN has already benchmarked. Correct refusal can be a successful outcome when the task requires the agent to decline an unauthorised action.
DN Repeatability Lab
Paste one line per scheduled trial. Use the same task ID for repeated executions of the same task under the same configuration and conditions. Do not pool different systems under one ID.
Format: task_id | accepted | human_minutes | technology_cost | elapsed_seconds
Accepted is 1 for a verified pass and 0 otherwise. Include unresolved trials as 0 for this conservative pass calculation and retain their status in your full ledger. Human minutes include all recorded human effort; this compact tool does not distinguish planned approval from rescue. Technology cost includes every retry within that trial.
This tool analyses entered records. It does not run an agent, check the truth of a pass label, infer causal failure reasons or certify reliability. Inputs are not transmitted or persisted by this script. Ordinary site analytics operate separately. No external dependencies are required.
Read the Fictional Example Correctly
The sample contains four task groups with three trials each. Ten of twelve trials pass, an observed acceptance rate of 83.3%. Eight pass with zero recorded human minutes, or 66.7% of all trials.
Two task groups pass all three trials. Only one group passes all three with zero recorded human effort. The brief group is consistently accepted, but one trial includes two human minutes. That could be a planned review rather than rescue; the compact record cannot tell us.
Total technology cost is 5.24 currency units. Human effort totals 23 minutes. At the fictional rate of 60 units per hour, included cost is 28.24, or 2.82 per accepted trial.
These observations describe the fictional records only. The tool’s all-pass group share is not a calibrated estimate of future pass^k. Unequal group sizes make that share especially difficult to compare.
What to Measure Beyond Acceptance
- Intervention-free accepted share: accepted trials with zero recorded human minutes divided by all trials.
- All-pass task groups: task groups in which every entered trial is accepted, with trial counts disclosed.
- Human effort: total minutes, split by role and reason in the full ledger.
- Cost per accepted trial: included spending across all trials divided by accepted trials.
- Timing distribution: median, high percentile and maximum elapsed time, with timeouts included consistently.
- Failure concentration: which task groups fail and whether failures cluster around shared dependencies.
Calling an accepted trial “autonomous” requires more than a zero in a time field. An operator may have prepared the environment, corrected inputs before the run or supplied undocumented context. Disclose those conditions.
A Perfect Sample Is Still a Sample
Ten successes in ten trials do not prove a 100% underlying success probability. Estimates need uncertainty, and common binomial methods depend on assumptions about independent observations and a stable probability.[4]
Pooling easy and hard tasks can hide meaningful differences. Shared failures can make nominal sample size overstate the amount of independent evidence. Report task-level results and uncertainty methods appropriate to the design rather than attaching a generic confidence label.
The DN tool intentionally reports descriptive observations without a confidence interval. It does not know whether the records are independent, representative or graded correctly.
The Signal
Look for accepted outcomes becoming more consistent while human effort and tail latency decline under matched conditions. Check that this improvement survives new task samples and that authority controls remain intact.
DN Alpha Thesis
Dependable delivery could become a stronger commercial differentiator than spectacular demonstrations. A system that produces slightly less impressive individual outputs may be more useful if its results are easier to verify, its exceptions are cheaper to handle and its performance is more predictable.
This is a conditional economic thesis. It weakens if customers value occasional breakthroughs more than consistency, if cheap verification and retries remove most variance costs, or if measured consistency fails to transfer into production.
DN Trial Ledger: What a Publishable Study Needs
Maintain a versioned record containing task and trial IDs, system configuration, input and initial-state references, acceptance rubric, outcome evidence, intervention category, human minutes, technology cost, elapsed time, retries, failure labels and exclusions.
Keep raw sensitive records private. Publish redacted examples and a machine-readable results file where permissions allow. State who ran the trials, who graded them and how disagreements were resolved.
A leaderboard should come after matched testing. Vendor documentation, demonstrations and self-reported success do not justify a measured DN ranking.
Methodology and Falsification
This article’s protocol is an editorial proposal. The calculator uses arithmetic on supplied trial records and treats each line as one trial. It does not attempt to distinguish model errors from harness, tool, infrastructure or grader errors.
The cost calculation includes entered technology costs and human minutes at one blended rate. It excludes unentered setup costs, overhead and downstream losses. Trials are evaluation repetitions; their accepted count should not be mistaken for unique commercial tasks completed.
To test the thesis, compare matched configurations and examine whether repeatability measures improve prediction of production acceptance, staffing needs or service quality beyond single-run performance. If they add little predictive value, DN should narrow the claimed benefit.
Unexpected failures should trigger investigation of the task, environment and grader as well as the agent. A stricter rubric is not automatically a better rubric if it rejects valid outcomes.
Continue Through the DN Toolkit
- Real-World AI Agent Benchmark: broader economic task coverage.
- Long-Horizon Agent Test: drift and recovery over extended work.
- Agent Model Efficiency: cost relative to verified outcomes.
- DN Agent Trust Dataset: authority and control evidence.
- Agentic Finance Guide: the wider execution and accountability stack.
Sources
- Anthropic: Demystifying evals for AI agents. Engineering guidance, published 9 January 2026.
- Towards a Science of AI Agent Reliability. Research preprint; interpret findings within its stated scope.
- ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions. Research preprint; not a financial safety certification.
- NIST/SEMATECH e-Handbook: confidence intervals for a binomial proportion. Statistical reference.
Frequently Asked Questions
What is AI agent repeatability?
It is consistency in meeting defined acceptance requirements across repeated executions under documented conditions.
Must repeated outputs be identical?
No. Different wording or valid execution paths can meet the same requirements. Evaluate the relevant outcome and constraints.
How is pass@k different from pass^k?
Pass@k concerns obtaining at least one successful solution across attempts. Pass^k concerns all repeated trials succeeding.
Does this article rank vendors?
No. DN has not completed a matched vendor study for this framework. The sample records are fictional.
What does zero recorded human effort mean?
It means the entered human-minutes field is zero. It does not establish autonomy if preparation or intervention was omitted.
How many trials are enough?
There is no universal count. The required sample depends on task coverage, target reliability, precision and dependence between observations.
Can these tests use real payments?
The proposed action scenarios use mock systems or sandboxes. This framework does not establish readiness to move live funds.
Does the tool execute or verify an AI agent?
No. It analyses reader-entered records and does not verify the underlying outcomes.
Does perfect observed performance guarantee future success?
No. A finite sample under specific conditions cannot guarantee future outcomes or performance in a different environment.
Disclosure: DN’s protocol and tool are editorial work. The illustrative dataset is fictional. No measured vendor ranking, security certification or investment recommendation is presented.
Related reading:
The Cost of Human Rescue: The Labour Bill Hidden Inside AI Automation
Model Intelligence vs Harness Intelligence: Why AI Agents Fail
We Tested Five AI Finance Systems. Here Is What the Evidence Actually Proves.
Who Gets Paid When Two AI Agents Disagree?
The DN Agent Credit Score: Measuring Whether Autonomous AI Can Repay
