AI Agent Acceptance Tests: How to Verify the Work Before You Pay

Research edition: 3 November 2026 · DN Research Export Acceptance Lab v0.1 · Synthetic local evidence

The agent reports that a research export is complete. The file opens. The JSON parses. Every row looks plausible. The buyer’s requested record is missing, one timestamp is stale and a number is expressed in the wrong unit.

All of those defects can survive a completion message. Some can survive a basic schema check. Other failures, such as a plausible but invented value, can survive a much stricter rubric unless the workflow also verifies the underlying evidence.

This article completes another part of DN’s agent evidence chain. The outcome receipt connects the task to billing and settlement records. The retry-boundary lab explores whether another attempt creates another effect. Acceptance testing asks a different question: did this delivered result meet the task’s agreed criteria?

The Signal: Completion, Structure and Acceptance Are Different Claims

HTTP semantics define what a 200 response means for the request. They do not supply the buyer’s task-specific research criteria. A successful response can deliver a result that is incomplete for the business decision the buyer needs to make.

Data engineering already has a discipline for making expectations explicit. Great Expectations documents individual data assertions and grouped expectation suites. JSON Schema provides structural validation and format vocabularies. DN’s contribution here is an inspectable example linking a fixed task rubric to an agent-output review, with controls showing exactly where that review stops.

One implementation detail deserves attention: in JSON Schema 2020-12, format annotation and format assertion are distinct. A schema containing a date or URI format does not justify assuming every validator configuration rejects every malformed value. The DN custom checker performs its own clock, coverage and exact-host checks. The downloadable structural schema is not a substitute for those checks.

DN Alpha Thesis: Protect the Acceptance Boundary From Moving Goalposts

DN calls the separation between the candidate output and the rules used to accept it the acceptance boundary. A model can help draft criteria, explain failures or propose a correction. The submitted result should not be able to relax its own rule and then declare success.

For example, a buyer asks for three named records, each with a current source reference and a value in USD cents. An export contains three rows, but two share the same identifier. A row-count check passes while required coverage fails. If the agent changes “three unique named records” into “three rows,” it has changed the task.

The same problem appears when freshness windows expand after a stale result arrives, missing sources become optional, or a numeric string is silently treated as a valid monetary amount. These may be reasonable changes in a renegotiated task. They should be explicit, authorized changes rather than hidden accommodations in the grader.

In a production workflow, pin the rubric version, its owner, the accepted task scope and the output being evaluated. Record exceptions separately. The browser lab illustrates this separation with a compiled rubric; it is not a secure enforcement point against someone who controls their own browser.

The Demonstration Task and Its Fixed Rubric

DN’s synthetic task requests asset-a, asset-b and asset-c, exactly once each, in any order. These are invented research records, not live asset prices or platform measurements.

• Identifiers: task, rubric and schema identifiers must match the fixed policy.
• Coverage: all three required unique record IDs must be present, with no extra record.
• Units: the currency must be USD and values must be integer cents between zero and 1,000,000,000,000.
• Time claims: generation and observation timestamps must be canonical UTC seconds within the inclusive 24 hours ending 3 November 2026 at 18:00 UTC. Observation claims cannot be later than generation.
• Source shape: references must use credential-free HTTPS on exactly research.example, with no nondefault port.

The reference clock and 24-hour window are synthetic choices for this example, not recommended defaults for every market or task. The .example namespace is reserved for documentation under RFC 2606. No source is fetched and no claim is authenticated.

Try the DN Research Export Acceptance Lab

Change a record ID, currency or timestamp in the synthetic example. Duplicate one ID while keeping three rows. Add a relaxed policy field to the candidate. Then try changing a value to another permitted integer: that can still pass because this checker has no factual oracle.

Fixed rubric JSONExample exportStructural schema26 synthetic fixturesExecuted resultsResults CSVMethodology and limits

The function bounds record arrays to one through 100 items to keep the demonstration manageable. The fixed task still requires exactly three named records. Unsupported fields and missing values are reported for review rather than silently discarded or converted.

What the 26 Local Cases Demonstrate

Coverage is more than row count

The duplicate-ID case contains three rows but misses one required unique record. It fails. Reordering all three valid records passes, because the rubric does not require a specific order. These controls matter together: the checker should reject missing coverage without inventing a restriction the buyer never requested.

Zero and “unknown” require different handling

A numeric zero passes the value rule. A missing source field, a numeric string and a boolean amount do not. Treating an absent value as zero can hide a data problem; rejecting every zero can also discard legitimate output. The agreed meaning and type belong in the rubric.

Time comparisons need a declared reference

The case exactly at the 24-hour cutoff passes. A claim one second before that boundary fails. Future observations, impossible calendar dates and observations after export generation also fail. These are checks of the reported values, not authenticated freshness measurements.

Plausible references and numbers can still pass without being true

The invented-source-path control uses a well-shaped reference on the allowed synthetic host. It passes without any fetch. The invented-value control changes a number to another value inside the permitted range. It also passes.

Those results are expected. They show where the rubric needs additional evidence if factual accuracy is part of the real task. A green result should retain its scope rather than become a general “verified” badge.

Acceptance Changes the Economics of an Agent Workflow

A cost-per-task claim is difficult to interpret until “task” has an acceptance definition. Counting responses, files produced or tools called can make output look cheaper while moving the verification burden to a human.

Consider a hypothetical cohort costing $100. Eighty requests return successfully, but only sixty outputs satisfy the agreed rubric. Dividing by successful responses gives $1.25; dividing by rubric-matching outputs gives about $1.67. Neither calculation establishes factual accuracy if the rubric does not check it. These numbers illustrate denominator choice, not DN-measured customer economics.

Then include review and exception costs. If every apparent success still needs a lengthy source audit, the operating benefit may be smaller than the model bill suggests. Measure that burden explicitly and keep the same cohort and period on both sides of the ratio.

This is why accepted-outcome economics and acceptance testing belong together. A payment scheme can define when a charge settles; the workflow still needs a defensible account of why an output qualified.

A Practical Acceptance Workflow

1. Before execution: identify the task, rubric owner, required output and evidence. Agree on exceptions and who may approve them.
2. At delivery: retain the candidate artifact and record which rubric version evaluated it.
3. Run bounded technical checks: structure, coverage, units and relevant time or source-reference rules.
4. Verify the substantive claims: use the appropriate authoritative sources, trusted comparisons or qualified human review for what the decision requires.
5. Record the acceptance decision: identify the authorized evaluator and any unresolved questions.
6. Apply commercial consequences separately: use the agreement for invoicing, retry allocation, disputes or further action.

For a first pilot, request one ordinary successful result, one incomplete result and one plausible but incorrect result. Include a legitimate edge case such as zero or a different ordering. A useful evaluator needs evidence about false acceptance and false rejection, not just examples chosen because they look good.

A probabilistic evaluator can assist with tasks whose quality cannot be reduced to deterministic rules. Its judgment needs its own assessment and escalation policy. Do not quietly replace the buyer’s definition with whichever metric the producing agent finds easiest to satisfy.

Methodology, Falsification and Limits

Evidence types: official standards and documentation were reviewed on 3 November 2026. The acceptance-boundary framework is DN’s editorial interpretation. The 26 test cases are invented records evaluated locally by the published custom function.

Results: all 26 declared expectations reproduce: six meet the fixed rubric and twenty do not. There were no model calls, paid requests, provider calls or source fetches. This is not a measured agent completion rate, a vendor comparison or a factual-verification benchmark.

Reproduce: download checker.mjs, fixtures.json and reproduce.mjs into one folder. Run node reproduce.mjs without dependencies or network calls. The result artifact records the executed checker’s SHA-256, which identifies source bytes rather than authenticating a publisher or timestamp.

Falsification: submit a reproducible case inside the supported scope where the function misclassifies one of its stated invariants. An invented in-range value passing is an expected limitation, not a detected factual claim. A production acceptance process needs its own independent evidence and enforcement.

Commercial disclosure and reuse: the Research Desk is DN’s paid service; its placement does not change test classifications. DN’s files are released under CC BY 4.0. Retain attribution and limits. Third-party material keeps its own terms. Send corrections with a case identifier and reproducible evidence; material changes should receive a new version.

Primary Sources

1. RFC 9110, HTTP 200 semantics.
2. JSON Schema 2020-12 validation and format vocabularies.
3. Great Expectations: Define Expectations.
4. RFC 2606, reserved documentation domains.
5. DN fixed synthetic rubric and executed local results.

Frequently Asked Questions

What is an AI-agent acceptance test?

It checks a delivered result against criteria agreed for a particular task. Criteria can include required records, units, freshness claims, source requirements and an authorized review process. Passing a technical check does not automatically establish factual truth or authorize payment.

Is a successful HTTP response enough to accept the task?

No. Request success, output structure, business-rule compliance and verified evidence answer different questions. The acceptance decision must use the agreed criteria for the actual task.

What does the DN lab check?

It checks a fixed synthetic export for structure, three required unique record IDs, claimed timestamps, USD integer units and source-reference syntax on one exact example host. It does not fetch sources or evaluate factual accuracy.

Can the submitted export change its own rubric?

No. The published checker’s rubric is compiled outside the candidate. Adding a policy field or changing task and rubric identifiers does not alter the rules. The browser demonstration itself is not a secure production enforcement boundary.

Does a current timestamp prove that the data is fresh?

No. The checker evaluates the timestamp supplied in the record against a fixed clock. It does not authenticate when the observation happened or whether the source supports the value.

Why can an invented value pass?

The rubric checks that a value has the permitted currency, integer type and range. It has no factual oracle for the true value. An invented in-range number can satisfy those rules.

Does the structural JSON Schema perform every acceptance check?

No. It is a structural guide. Format assertion depends on validator behavior and configuration. The published custom checker separately evaluates the fixed clock, exact host and required unique coverage.

What were the test results?

All 26 local synthetic cases matched their declared expected classifications: six met the fixed rubric and twenty did not. These are tests of a small checker, not an observed AI-agent success rate or vendor benchmark.

Can I use the lab on live commercial data?

Use only synthetic or redacted examples in the published format. Never paste credentials, personal information or live commercial exports. The checker itself makes no requests and stores no input; ordinary website analytics are separate.

How do I reproduce and reuse the research?

Download checker.mjs, fixtures.json and reproduce.mjs into one folder and run node reproduce.mjs. DN’s files are licensed CC BY 4.0; retain attribution and limitations. Third-party documents keep their own terms.

Evidence cut-off: 3 November 2026. DN Research Export Acceptance Lab v0.1. A rubric match does not establish factual truth or authorize payment.