The Signal: Benchmark Scores Are Not the Same as Economic Utility

AI benchmarking has progressed rapidly.

SWE-bench evaluates whether systems can solve genuine software-engineering issues. Its Verified dataset contains 500 tasks reviewed by humans for clarity and solvability.

OSWorld moves evaluation into real computer environments rather than static prompts. Its newer generations increasingly emphasize longer, more realistic application workflows.

Princeton's τ-bench family evaluates agents that must converse with users, invoke tools and follow policies inside realistic domains such as airline support.

METR measures task-completion time horizons by comparing agent success against the amount of time expert humans require to complete tasks.

These are meaningful advances.

But each benchmark measures a slice of useful work.

The Missing Unit: Paid-Quality Completion

Work has several characteristics that ordinary benchmark success rates can hide.

A research report can contain the requested facts and still be unusable because its sources are wrong.

A sales prospect list can contain 100 rows and still be worthless because half the contacts are irrelevant.

A spreadsheet can calculate correctly while silently corrupting the formatting or formulas elsewhere.

A customer-service agent can eventually reach the correct outcome but consume more time and money than a trained employee.

An agent can successfully finish a task once and fail the next four times.

That means binary completion is insufficient.

DN proposes a higher standard:

The DN 100-Task Real-World Agent Benchmark

DN-RWAB 1.0 divides 100 tasks across ten categories representing commercially meaningful knowledge work.

What the 100 Tasks Should Actually Look Like

A useful benchmark should avoid toy requests that can be solved by pattern matching.

Each DN task should contain realistic constraints, imperfect information and a verifiable outcome.

DN's Seven-Dimensional Scoring Model

Every benchmark run should generate more than a pass or fail.

Try the DN Economic Task Score

DN Economic Task Score Bands

The Benchmark Must Penalize Lucky Runs

One successful demonstration is not reliability.

Agent systems are probabilistic, and multi-step workflows create more opportunities for small differences to compound.

DN-RWAB therefore proposes at least three independent runs per task for baseline testing.

High-value or safety-sensitive tasks should eventually use more.

Why Repeatability Changes the Leaderboard

Imagine two agents.

Agent A successfully completes 85 of 100 tasks on its best run.

Agent B completes 78.

That headline makes Agent A appear better.

Now run each task five times.

If Agent A succeeds unpredictably while Agent B produces the same correct result almost every time, the commercial conclusion changes.

Businesses often care more about predictable 78% performance than an unstable system capable of occasionally reaching 85%.

Public Benchmarks Already Show Why Harnesses Matter

Agent evaluation measures more than the underlying model.

The same model can perform differently depending on its tools, prompts, memory architecture, environment, retry strategy and agent scaffolding.

SWE-bench explicitly distinguishes model comparisons from broader agent-system submissions. Princeton has also removed compromised benchmark results when test-set leakage was identified in a τ-bench scaffold.

That is important.

DN-RWAB Should Score the System, Not the Brand

Every leaderboard entry should record:

• model and exact version;
• agent framework or harness;
• tools available;
• maximum steps;
• reasoning or compute configuration where disclosed;
• external search access;
• memory configuration;
• human interventions;
• retries;
• total tokens or compute usage where measurable;
• API and tool cost;
• wall-clock completion time;
• test date;
• task version.

Without these details, a leaderboard risks comparing fundamentally different systems under one model name.

The Cost per Successful Task Is More Useful Than Token Price

Cheap tokens do not necessarily create cheap work.

Suppose Agent A costs $0.20 per attempt but succeeds only 30% of the time. Its effective model cost per successful task is approximately $0.67 before human rescue costs.

Agent B might cost $0.50 per attempt but succeed 90% of the time, producing an effective cost near $0.56 per successful completion.

The supposedly more expensive agent is economically better.

This is why DN proposes:

Human Intervention Must Be Priced

One of the largest hidden costs in agent deployments is human rescue.

An agent may cost pennies to run but require ten minutes of professional review after every task.

At scale, the review cost can dominate inference.

DN-RWAB therefore treats human involvement as part of the economic cost rather than pretending it is free.

The Economic Replacement Ratio

DN proposes another metric for the benchmark:

An Economic Replacement Ratio of 1 means agent and human cost are approximately equivalent.

A ratio of 5 means the verified agent workflow costs roughly one-fifth of the competent human baseline.

But cost advantage alone is insufficient.

A cheap workflow producing unusable work has no meaningful replacement value.

Example: A $100 Human Task vs a $5 Agent Run

Consider a market-research task that a competent freelancer would charge approximately $100 to complete.

The agent workflow costs $5.

At first glance, the economic replacement ratio appears to be 20×.

But suppose the agent produces a usable answer only half the time and requires 20 minutes of human checking and correction.

The genuine cost advantage could be dramatically lower.

Benchmarking therefore has to measure the entire production loop.

Ten Research Tasks Humans Actually Pay For

Ten Finance and Accounting Tasks

Ten Sales Tasks

Ten Marketing and Content Tasks

Ten Software and Technical Tasks

Ten Customer Operations Tasks

Ten Commerce and Procurement Tasks

Ten Administrative Tasks

Ten Data and Spreadsheet Tasks

Ten Agentic Finance Tasks

The Hardest Part Is Ground Truth

Real-world tasks are more difficult to benchmark than multiple-choice questions because the correct result may contain judgment.

DN-RWAB should therefore use three types of evaluation.

The Human Baseline Matters

An agent cannot be described as economically competitive without a comparison point.

Every task should therefore estimate:

• competent human completion time;
• competent human cost;
• expected human error range where measurable;
• minimum acceptable professional quality.

This does not mean humans must achieve 100%.

In fact, comparing agents against realistic human performance rather than an imaginary perfect worker makes the benchmark more useful.

Task Horizon Is Becoming a Critical Metric

METR's task-horizon work provides an important insight: agent capability is not only about whether a system can do something, but how long and complex a human-equivalent task it can complete reliably.

Real employment contains tasks ranging from two-minute lookups to projects that consume days.

DN-RWAB should therefore record estimated competent-human completion time for every task.

Why OSWorld 2.0 Matters to This Thesis

The direction of recent benchmark design already points toward longer real-world workflows.

OSWorld 2.0 introduced 108 long-horizon computer-use workflows whose human completion time had a median of roughly 1.6 hours.

That is materially closer to professional work than a short GUI task.

Its published research still found frontier systems struggling with the full workflows, especially when success required tracking hidden state, changing information and multiple constraints.

The conclusion is not that agents are incapable.

It is that realistic work exposes failure modes that simpler tasks can miss.

The Benchmark Should Include Failure Recovery

Humans encounter broken links, missing data, failed APIs, ambiguous requests and changed circumstances.

Real agents will too.

A practical benchmark should therefore deliberately include recoverable failures.

Examples:

• a supplier page becomes unavailable;
• a spreadsheet contains an unexpected column;
• a user provides conflicting instructions;
• a tool call fails temporarily;
• a booking option disappears;
• a transaction simulation fails;
• a source contradicts another source;
• required information is missing.

A robust agent should know whether to retry, use another route, ask for clarification, escalate or stop.

The Benchmark Should Penalize Confident Guessing

One particularly expensive agent failure is plausible fabrication.

An agent that asks for missing information may seem less autonomous than one that invents an answer and proceeds.

Economically, the cautious agent may be far more useful.

DN-RWAB should therefore reward appropriate abstention.

If success requires information the system does not possess, the correct behavior may be to stop and request it.

Agents Should Be Measured on Net Labor Saved

Gross automation can be misleading.

A workflow may eliminate 60 minutes of manual work while creating 25 minutes of checking, correction and system maintenance.

The real productivity gain is smaller.

This makes it possible to compare systems that automate different portions of the same workflow.

The DN Agent Economic Frontier

Over time, the most revealing visualization may not be a single ranking.

It may be a frontier.

Plot each tested system by:

• paid-quality completion rate;
• cost per successful task;
• human intervention minutes;
• median task horizon;
• repeatability;
• failure severity.

Different systems may dominate different parts of this frontier.

One agent might excel at short low-cost administrative work.

Another may be expensive but capable of completing long technical projects.

A third may be exceptionally reliable in financial workflows where correctness matters more than raw speed.

Why a Single “Best AI Agent” Is Usually the Wrong Question

Agent performance is task dependent.

A coding agent should not automatically be considered superior to a customer-service agent because it performs better on a software benchmark.

DN-RWAB should therefore publish:

• overall score;
• category scores;
• task-horizon distribution;
• cost per successful task;
• reliability;
• human-intervention requirement.

Readers can then select systems according to the work they actually want performed.

Proposed DN Leaderboard Structure

DN will not fabricate benchmark scores before controlled runs are completed. Initial leaderboard cells should remain explicitly unscored until reproducible test data exists.

How DN-RWAB Differs From Existing Agent Benchmarks

For Individuals: Measure What You Can Stop Paying Someone Else to Do

For freelancers, creators and professionals, the useful question is not how many benchmarks a model tops.

It is which tasks can now be delegated without increasing mistakes.

Start by documenting recurring paid or time-consuming work:

• research;
• data cleaning;
• report preparation;
• prospecting;
• bookkeeping preparation;
• content repurposing;
• administration.

Test the agent repeatedly and compare the net labor saved.

For Businesses: Build a Private Version of the Benchmark

Public benchmarks should be treated as evidence, not production guarantees.

A company deploying agents should construct a private evaluation set from its own recurring work.

Twenty authentic internal tasks may reveal more about deployment readiness than hundreds of unrelated public benchmark tasks.

The strongest approach is therefore:

1. use public benchmarks to shortlist systems;
2. use DN-RWAB-style economic metrics to compare commercial usefulness;
3. use private internal tasks before granting production authority;
4. continue measuring live outcomes after deployment.

For Agent Developers: Optimize for Completed Economic Work

DN-RWAB creates a different engineering target.

Instead of optimizing only for tokens, benchmark percentage or tool-call accuracy, developers would optimize for:

• correct outcome per dollar;
• completed work per hour;
• low human rescue;
• high repeatability;
• safe abstention;
• reliable recovery;
• professional output quality.

Those metrics map more directly to adoption.

The Economic Threshold for Agent Adoption

An agent becomes commercially interesting when four conditions intersect:

Improving model intelligence addresses only the first condition.

The commercial agent economy depends on all four.

DN Methodology

Limitations

No 100-task benchmark can represent the full labor market.

Work quality can be subjective, human cost varies by geography, and some jobs depend on tacit knowledge, relationships or physical actions that digital agents cannot reproduce.

Benchmark systems can also improve specifically against known datasets.

DN should therefore rotate private holdout tasks, publish methodology transparently and avoid presenting one aggregate score as universal intelligence.

Primary Sources and Benchmark References

Frequently Asked Questions

Disclosure

Decentralised News develops independent research frameworks, indices, benchmarks and decision tools covering AI, crypto and agentic finance. Some Decentralised News pages may contain affiliate or commercial relationships. These relationships do not determine benchmark inclusion, methodology, scores or editorial conclusions. The DN-RWAB framework is an original research methodology and does not constitute financial, employment or investment advice.