The Agent Model Efficiency Index 2027
Which AI model produces the lowest total cost per verified successful task? DN-A-MEI replaces token-price comparisons with an outcome-adjusted measure covering retries, tools, latency and human rescue.
What Matters
The cheapest model is not the model with the lowest token price. It is the model-agent combination that completes a defined task reliably at the lowest total production cost. DN-A-MEI measures model spend, tool fees, retries, latency and human rescue, then divides that total by verified successes. Model selection should therefore happen per workflow, not through one universal leaderboard.
DN Evidence Block
- API providers publish token prices, but token price alone does not disclose completion rate, retries, tool spend or human intervention.
- SWE-bench separates model and agent-system submissions and uses executable tests, demonstrating why the scaffold must be reported with the model.
- Terminal-Bench reports resolution rate alongside cost and tokens, providing an important precedent for outcome-adjusted evaluation.
- Recent benchmark research warns that flawed tasks, evaluators and reward designs can materially inflate reported performance.
- DN does not claim a current model winner without matched tasks, fixed scaffolds, repeated trials and auditable cost logs.
The DN Thesis: Tokens Are Inputs, Success Is the Product
Most model comparisons stop at input price, output price, speed or a public benchmark score. None of those is the economic unit a business buys. A business buys a resolved ticket, reconciled invoice, correct research brief, deployed code change or completed customer request.
This creates the Token Price Illusion: a model can appear inexpensive while generating more failed runs, longer outputs and more human review. A premium model can be cheaper if it completes more tasks on the first attempt. The reverse is also possible when a smaller model handles a narrow, well-specified workflow reliably.
What DN-A-MEI Measures
Verified completion
The task must satisfy an executable test, deterministic rubric or documented human-review standard. Plausible-looking output is not success.
Complete cost
Input, cached input, output, reasoning, search, tool calls, compute, retries and human intervention belong in the numerator.
Repeated performance
Pass@1 shows first-attempt utility. Repeated trials reveal variance, while pass^k exposes whether the system can succeed consistently.
| Dimension | Weight | Measurement | Why it matters |
|---|---|---|---|
| Cost per verified success | 35 | Total production cost divided by accepted outcomes | Direct unit-economics signal |
| First-pass completion | 20 | Accepted tasks on the first run | Limits retry and latency penalties |
| Repeatability | 15 | Variance across matched repeated runs | Separates demos from production |
| Latency to verified result | 10 | End-to-end wall-clock time | Captures operational delay |
| Human rescue burden | 10 | Minutes and cost of intervention | Prevents hidden labor from disappearing |
| Tool-call efficiency | 5 | Useful calls versus invalid, duplicated or unnecessary calls | Captures agent execution quality |
| Auditability | 5 | Trace completeness, versioning and reproducibility | Makes the result defensible |
DN Cost per Successful Task Calculator
Enter observed values from your own workflow. The calculator estimates outcome-adjusted cost and exposes how an apparently cheap model becomes expensive when failures require retries or human rescue.
Why a Universal Model Leaderboard Would Mislead
A model does not act alone. System prompts, tool descriptions, retrieval, memory, context selection, temperature, retry logic and approval gates all affect outcomes. Comparing Model A inside a mature coding agent with Model B inside a minimal loop measures two systems, not two models.
Public benchmarks remain useful for narrowing candidates. They are not a substitute for private evaluations on the task distribution that creates value for the buyer. A coding benchmark cannot establish customer-support efficiency. A browsing benchmark cannot establish accounting reliability.
The Minimum Viable Model Bake-Off
- Define the economic task: write an unambiguous start state, permitted tools and accepted end state.
- Build at least 30 representative cases: include normal, edge and adversarial examples. Higher-stakes choices need larger samples.
- Freeze the harness: use the same prompt policy, tools, retry ceiling and context budget.
- Log every cost: include model tokens, paid tools, compute, search, storage and review time.
- Use blinded verification: evaluators should not know which model produced an output where practical.
- Repeat the test: one pass hides variance. Report confidence intervals and failure categories.
- Route by task: keep premium models for cases where their incremental completion rate exceeds their incremental cost.
Decision Matrix: Choose by Workflow, Not Reputation
| Workflow | Optimize first | Avoid | Best test design |
|---|---|---|---|
| High-volume classification | First-pass accuracy and low unit cost | Paying for unused reasoning depth | Large labeled holdout set |
| Customer service | Resolution, policy compliance and escalation accuracy | Judging only response fluency | Multi-turn executable simulations |
| Coding agent | Tests passed, regression safety and repair cost | Mixing different scaffolds without disclosure | Repository tasks with hidden tests |
| Research agent | Claim accuracy, source quality and coverage | Rewarding verbosity as completeness | Time-stamped evidence packets |
| Financial operations | Correctness, audit trail and loss containment | Autonomy without approval thresholds | Sandboxed ledger and exception set |
| Computer-use agent | End-state completion and harmful-action rate | Counting partial navigation as success | Resettable environments with state checks |
The Model Routing Opportunity
The economic optimum is often a portfolio, not a single model. A smaller model can classify or draft routine cases, while a stronger model receives ambiguous cases. The router itself must be evaluated because false escalation increases cost and false non-escalation increases failure.
A useful routing rule compares the additional cost of the stronger model with the expected reduction in failure and rescue cost. If a $0.40 premium reduces expected downstream loss by $2.00, escalation is economically rational. If it saves only $0.05, it is not.
Current Pricing Is an Input, Not a Verdict
OpenAI, Anthropic and Google publish model-specific API prices, with separate treatment for input, output, caching and some tools. These prices change and may vary by processing tier, context length or modality. DN therefore links to live pricing pages rather than freezing a universal cheapest-model table into the index.
For every test run, record the exact model identifier, provider, region, service tier, date, input tokens, cached tokens, output tokens, tool charges and credits. Promotional prices and free tiers should be reported separately from steady-state production economics.
Methodology, Limitations and Falsification
DN-A-MEI v1.0. The index evaluates a model-agent configuration on a declared workload. Scores may be compared only when the task set, harness, tool access, retry policy, evaluator and pricing window are materially equivalent.
Proposed normalization: the 35-point cost component uses the inverse of cost per verified success relative to the lowest eligible system in that workload. All remaining dimensions use declared, versioned scoring rules. A system must complete a minimum sample and retain auditable traces to qualify for ranking.
Limitations: this publication is a methodology release, not a claim of hands-on comparative testing. Calculator outputs depend on user inputs. Public benchmark results can be affected by contamination, task quality, harness differences and evaluator design.
What would prove the thesis wrong? If token price consistently predicts total verified task cost across diverse workflows after controlling for scaffolds, retries and labor, the broader framework would add little value. DN expects the opposite and will test this in future matched evaluations.
Update cadence: pricing inputs monthly, methodology after material benchmark research, and matched DN leaderboards quarterly once the test dataset is released.
Change log: v1.0 establishes the outcome-adjusted cost formula, seven-dimension score and interactive calculator. Corrections can be submitted through DN Contact.
Commercial and Editorial Disclosure
No provider receives a higher score because it sponsors DN or offers an affiliate programme. This edition contains no model-provider affiliate ranking. Future sponsored benchmark runs must use the same published methodology, disclose funding and preserve DN's right to publish unfavorable results.
Frequently Asked Questions
What is cost per successful task?
It is the total cost of attempts, tools, infrastructure, retries and human rescue divided by the number of verified successful outcomes.
Why is token price not enough?
Token price excludes completion rate, output length, retries, tool calls, latency and human intervention.
Which AI model is most efficient in 2027?
There is no defensible universal winner. Efficiency depends on the workflow, scaffold, success standard, pricing and failure cost.
Can public benchmark scores be used?
Yes, for candidate screening and regression tracking. Production selection still requires matched tests on representative private tasks.
What counts as a successful task?
A task must satisfy a predeclared executable test, deterministic rubric or documented human-review standard.
How many test tasks are enough?
Thirty cases can support an early directional comparison, but consequential decisions require broader samples, repeated runs and uncertainty reporting.
Should retries be allowed?
Yes, if the retry policy is fixed and every retry cost is included. Report first-pass completion separately.
How should human review be priced?
Multiply intervention minutes by a fully loaded hourly labor rate and include that cost in the numerator.
Can one model handle every workflow?
It can, but task-based routing often reduces cost by reserving expensive models for difficult or high-risk cases.
How frequently should model efficiency be retested?
Retest after material model, price, prompt, tool or workflow changes and on a scheduled quarterly cadence.
