What Matters
A useful agent dataset must say what was observed and what remains unknown. DN’s Managed Runtime Evidence Dataset v0.1 publishes five documentation records, six proposed runtime tests and machine-readable fields for future measurements. It contains no executed provider benchmarks or winner. The value of this release is an inspectable baseline that prevents unsupported claims from looking like measured results.
DN Evidence Block
- Last verified: 19 October 2026. The 2027 title is an editorial planning horizon, not a future observation date.
- Scope: Five documented architecture paths across OpenAI, AWS, Google, Microsoft and a self-hosted LangGraph reference.
- DN contribution: An original versioned JSON baseline, six proposed test contracts and local data-validation fixtures.
- Limits: Zero executed provider benchmark runs. No live payment, migration, model inference or cloud deployment was performed for this release. The local validator tests record integrity, not runtime safety.
- Prepared by: Decentralised News research desk. No independent reviewer is claimed. Submit a correction.
Start with a ledger of claims
Comparison pages often place an observed feature, a vendor promise and a benchmark result in adjacent cells with the same visual weight. Readers can mistake the table's neatness for comparable evidence. A dataset should make the difference machine-readable as well as visible in prose.
Version 0.1 starts with source observations. Each record names an architecture path, links a primary document, states a review date and leaves performance fields null. The source observation can support a narrow claim about documented design. It cannot support a task-completion ranking, a security classification or a claim that one provider is cheaper.
Download the complete versioned JSON dataset. The record schema distinguishes documentation from future execution. DN’s original observations and test descriptions are available under CC BY 4.0; vendor documentation and third-party material retain their own rights.
The first five records
| Record | Observed documentation boundary | Runtime measurements |
|---|---|---|
| OpenAI Agents API / SDK | Documentation separates managed agent sessions from application-controlled SDK execution. Primary source | Unmeasured; all metric fields null |
| Amazon Bedrock Managed Agents powered by OpenAI | AWS labels BMA public preview and separates conversation and execution-environment lifecycle. Primary source | Unmeasured; all metric fields null |
| Google ADK / managed Agent Runtime | ADK deployment documentation describes sending application code and dependencies to a managed runtime. Primary source | Unmeasured; all metric fields null |
| Microsoft Foundry Agent Service | Documentation distinguishes prompt-agent configuration from hosted custom-code agents. Primary source | Unmeasured; all metric fields null |
| Self-hosted LangGraph reference | Documentation distinguishes thread checkpoints from cross-thread stores. Primary source | Unmeasured; all metric fields null |
Read the empty cells correctly
The dataset stores an empty benchmarkRuns array and null measurement fields. That is intentional. “Unmeasured” is different from “zero failures” and from “a failed test.” A task never attempted cannot earn a success rate, and a cost-per-outcome ratio cannot be computed without both the full cost and the accepted-outcome denominator.
A later observation should identify the exact model, harness, tool versions, environment, dates, task family and acceptance oracle. The model name alone is insufficient. The surrounding runtime can alter tool selection, state recovery and approval behavior even when the underlying model is unchanged.
The six proposed runtime tests
| Test | Boundary | Required independent evidence |
|---|---|---|
| RT-01 | Bounded completion | An independently checked accepted outcome under pinned task and policy. |
| RT-02 | Lost response after commit | Stable action identifier, authoritative target state and absence or presence of duplicate effects. |
| RT-03 | Scope narrowing | A prohibited child request is rejected without an external effect. |
| RT-04 | Revocation | A timestamped grant is rejected after its declared effective revocation boundary. |
| RT-05 | State transfer | A resumed synthetic task preserves required state and authority with a documented mapping. |
| RT-06 | Full cost accounting | Spend for all attempts, human work and an independently graded accepted-outcome denominator. |
The cases are DN’s proposed research design, not executions or a complete security suite. Start with synthetic inputs and isolated targets. Record unauthorized effects separately from quality, latency and cost. A simulated policy check should not be relabelled as a production authorization result.
What local validation does prove
The accompanying validator checks that the baseline has unique record identifiers, primary HTTPS source links, explicit evidence types and limitations, null unmeasured metrics, and no disguised benchmark runs. It also rejects synthetic fixtures that claim execution without the fields needed to interpret it. The public validation receipt states its exact scope and the checks actually run.
Read the local validation receipt. Passing these checks means the record contract behaved as specified for the evaluated fixtures. It does not mean a provider passed RT-01 through RT-06, that an agent is safe, or that DN has exercised a cloud deployment.
DN Runtime Evidence Collection Planner
Record your own evidence for one pinned workflow. These are your declarations, not DN findings, a safety score or a scan. Unknown is the default. Do not enter credentials or private task content.
All boundaries start unknown. The surrounding table explains what evidence to collect; this worksheet does not inspect a provider.
Runs locally with no account, wallet, model call or storage of your answers. A downloaded worksheet contains your selections. Normal DN site analytics are separate.
How to contribute a useful run
- Agree a bounded task family, versions and acceptance rules before running.
- Record environment and permissions, including every external system that can change.
- Retain input/output structure and an independent result oracle without secrets or personal task content.
- Include every attempt, retry and unresolved result; explain exclusions.
- Submit costs with their accounting scope and currency. Do not present a token bill as a full deployment cost.
- State which conclusions the run supports and which remain unknown.
A contribution is not automatically accepted as independent evidence. DN should inspect reproducibility, licensing, privacy and conflicts of interest before adding a new observation. Vendor-supplied runs need clear attribution; a sponsored study needs its protocol and commercial relationship disclosed.
Commercial use without selling the answer
A versioned, inspectable record can support procurement research, scoped implementation comparisons and eventually a documented data feed. Those are opportunities, not products or subscription tiers claimed to be live today. Benchmark sponsorship can fund work only if the method, exclusions and adverse outcomes remain publishable.
Use the Runtime Portability Index to identify migration boundaries and the Agent Cost Stack to define the economic denominator. The Agent Trust evidence system remains separate: this baseline does not change its existing scores or classifications.
DN Alpha Thesis
The defensible asset is a growing set of evidence that other researchers can inspect, reproduce and challenge. Versioning and explicit unknowns make a modest dataset more useful than a confident ranking without comparable runs. The next release earns its claims by executing the protocol, not by filling empty cells with estimates.
Methodology, limitations and change log
Version 0.1, 19 October 2026. Five primary-documentation observations and six original proposed tests. Provider benchmark runs: zero. Local fixtures validate the record contract only. Documentation is a dated source review, not an immutable product guarantee. A later dataset revision must preserve the original release and publish its source and methodology changes.
Report a correction or propose an isolated run through DN’s contact page. Include the record/test identifier, exact version and evidence. Never submit credentials, recovery phrases, private customer data or unrestricted payment access.
Primary sources
- OpenAI agents architecture
- AWS Bedrock Managed Agents documentation
- Google ADK managed deployment
- Microsoft Foundry Agent Service
- LangGraph persistence
Frequently asked questions
Has DN benchmarked the five providers in this release?
No. Version 0.1 contains five documentation observations and six proposed test cases. The benchmarkRuns array is empty.
What do null metrics mean?
No measurement is supplied. Null is not zero cost, zero failures, a failed test or a passing result.
Can these records change Agent Trust classifications?
No. This is a separate baseline. Existing Agent Trust profiles and classifications are not altered.
Can I reuse the data?
DN’s original observations and test descriptions are offered under CC BY 4.0. The license does not relicense linked vendor documents or third-party traces.
Disclosure: Educational research, not investment advice or a security certification. No affiliate or sponsored placement appears in this edition. DN may discuss scoped research services; payment cannot purchase a finding. Corrections should identify the affected claim, observation date and supporting source.
