Decentralised News · Agentic Finance · Decision research
By Heath Muchena · Source review 2026-11-11 · Instrument version 1.0.0
What Matters
A human approval button proves that someone could intervene. It does not show whether intervention improved the decision. DN’s Human-Override Effectiveness Score records corrections, new errors and unresolved cases on the same adjudicated work. It helps a buyer distinguish useful oversight from a queue that adds delay while leaving the important mistakes untouched.
“Human in the loop” describes an arrangement. To assess its value, a buyer needs to know what the person sees, what they can change, how much time they have and whether their final decisions are correct. Approval counts answer none of those questions.
The practical test is a transition record: when the AI is wrong, does the human correct it; when the AI is right, does the human preserve the correct result? Those two measurements reveal more than the percentage of items reviewed or the number of times an override button was used.
DN Evidence Block
Sources reviewed: 2026-11-11 · Evidence type: A primary systematic review of human-AI performance and the NIST AI Risk Management Framework, followed by an original DN evaluation protocol.
Scope: Effectiveness of an assisted review process on a defined case set. No conclusion is drawn about whether a particular task legally requires human review.
- A Nature Human Behaviour meta-analysis covering 106 experiments found that human-AI combinations improved on humans alone on average, yet underperformed the better of the human or AI alone. Its studies came from 2020 to mid-2023; it is not a forecast of current deployments.
- The NIST AI RMF Core supports documented evaluation, appropriate independent assessment and monitoring. It supplies a risk-management framework, not a passing score for a human approval process.
Not established by this work: DN has not run the proposed human-review study. The numerical example is synthetic and deliberately balanced.
Original DN instrument · version 1.0.0
DN Human-Override Effectiveness Score
Calculate correctness transitions in a completed, independently adjudicated sample. The outputs describe the sample; they are not a universal oversight score.
Enter your assumptions or load a labelled example. Required blank numbers stay missing; unknown evidence remains unknown. Use nonpersonal labels only. This instrument makes no network requests and uses no persistent browser storage.
Use your own inputs
No current result. Complete inputs or load an illustrative example.
The tool assumes independent adjudication and correctly recorded transitions. It omits uncertainty intervals, case weighting, severity, capacity and cost. Unresolved cases are not silently assigned a result.
1. The person sees the AI's answer before forming a judgement
A polished recommendation can become the starting point for the review. The person checks whether it looks plausible instead of independently asking what the evidence supports. The interface may make this more likely if it places a confident verdict above the source material or hides the contrary evidence behind another screen.
Test the arrangement with a bounded case study. For some cases, ask the reviewer to form an initial judgement from the permitted evidence before seeing the AI’s answer. Record that judgement separately from the assisted final decision. For other cases, preserve the normal workflow. Assign cases or reviewers in a documented way and avoid giving a person the same case twice if recognition would influence the result.
This does not mean every production decision needs a duplicated blind review. The study asks whether the interface changes useful error detection. A slower independent reading might be worthwhile for a consequential exception and unnecessary for a routine reversible item.
Keep reviewer reasoning tied to specific evidence. “The AI explanation looked thorough” is not the same record as “the contract excludes this use in clause X.” The difference determines whether a later operator can understand and challenge the decision.
2. Approval is recorded even when the evidence was unavailable
A person can click approve without access to the material needed to decide. A source may be missing, a calculation may be untraceable or the reviewer may lack the expertise to interpret the exception. In that situation, adding a person to the route does not supply the absent evidence.
Give the review record an explicit unresolved state. Specify the missing item, its owner and the next action required to resolve it. A temporary operational block can be justified without labelling the underlying output wrong. Conversely, a routine approval should not imply that unavailable evidence was inspected.
For a study, record what the person could actually access at the time. A complete dossier assembled afterwards can make an earlier decision appear more informed than it was. Timestamped source availability, interface state and the version of the AI recommendation help preserve the true decision context.
Separate expertise from authority. A person may understand why the recommendation is wrong but lack permission to change a payment, a public statement or a supplier commitment. The workflow must define an escalation route with someone who can make the required decision. Otherwise human review records an objection that cannot affect the outcome.
3. The queue gives reviewers too little time to investigate
Review capacity is part of the control. If a process depends on reading supporting records but schedules only enough time to glance at a summary, its design contains a mismatch. Measure review time, queue age and the work displaced by escalations alongside decision quality.
Use task-specific time observations rather than assuming one average fits every case. A clear routine item and a contradictory exception can have very different needs. Report how many cases arrived with incomplete evidence and how long they waited for a resolution outside the reviewer’s control.
A sensible experiment changes one operational feature at a time: queue allocation, presentation of evidence or the ability to request clarification. If the model, interface and staffing all change together, it becomes difficult to identify which part improved the result.
Do not convert an arbitrary time target into a quality threshold. A rapid decision may be well founded; a long review may still be wrong. Time explains capacity and cost. Adjudicated correctness explains decision quality. Both are needed when deciding whether the oversight arrangement can support a higher workload.
4. Overrides are counted without checking their direction
An override can correct an AI mistake or spoil a correct answer. Counting the number of overrides rewards activity without establishing value. DN’s instrument therefore separates four states: wrong AI decisions corrected, wrong decisions left wrong, correct decisions preserved and correct decisions made wrong.
Consider an illustrative set containing 100 AI errors and 100 correct AI decisions. Humans correct 60 of the errors but overturn three correct decisions incorrectly. The final process has 157 correct decisions out of 200, or 78.5%, compared with the AI’s 50% on this selected set. The net gain is 57 correct decisions.
That example deliberately oversamples AI errors to make the arithmetic visible. It cannot be used to estimate a real production accuracy or return on investment. A production estimate needs the actual case mix or a documented weighting method, adequate observations and uncertainty analysis.
The calculator reports the selected sample, with unresolved cases visible outside it. It does not score the person’s diligence or determine whether oversight should be removed. Add a case-level severity field: correcting one minor formatting issue and creating one serious unsupported commitment should not be treated as an acceptable trade merely because the counts cancel.
5. The organisation measures approval coverage instead of final outcomes
A dashboard saying “100% reviewed” establishes process coverage only if the review event itself is trustworthy. It still leaves the result open. Check the delivered decision after the human action, including whether the chosen change actually reached the destination system.
Define independent adjudication before running the trial. For a calculation, that may be a reproducible recomputation. For a contractual condition, it may require an appropriately qualified reviewer and the actual governing record. Where reasonable experts disagree or the evidence remains incomplete, retain that state instead of manufacturing ground truth.
Compare the assisted process with the relevant baseline on the same task. An AI-only result, a human-only result and an assisted result answer different questions. A historical human team facing easier cases is not a fair control for a new AI-assisted team handling difficult exceptions.
Use the results to change the control precisely. Improve source access if people cannot find the evidence; revise escalation authority if objections cannot be acted on; change queue allocation if complex cases are rushed. The strongest outcome is a documented reduction in material mistakes under a workable operating model, not a larger volume of approvals.
Methodology, limitations and corrections
DN uses a transition-based evaluation design, informed by the cited meta-analysis and NIST framework. The five mechanisms are operational hypotheses to inspect in a particular workflow. The 200-case example is synthetic and intentionally balanced; none of its percentages describes a deployed product. A real study should document sampling, reviewer training, source access, randomisation where used, baseline comparability, adjudication and treatment of disagreement. Report severity and review time alongside the calculator outputs, and retest when the interface or delegated authority changes.
For corrections, identify the claim, source and date when contacting Decentralised News. A new record-creation timestamp does not refresh its underlying evidence.
Change log: version1.0.0, 2026-11-11. Initial documentation review, original instrument and labelled examples.
Frequently asked questions
Does a low override rate mean people are rubber-stamping?
Not necessarily. The AI may be correct on most cases. Examine independently identified AI errors and whether people correct them; raw override frequency lacks that denominator.
Can this score justify removing mandatory human review?
No. The tool evaluates a chosen sample. Applicable legal, contractual and organisational duties are separate and require their own assessment.
Why count human-induced errors?
Assisted review can correct some mistakes while introducing others. Preserving correct AI decisions is part of the control’s value and should remain visible.
Primary sources
- When combinations of humans and AI are useful: A systematic review and meta-analysis. Study-specific comparison of human-AI combinations with human and AI baselines. Reviewed 2026-11-11.
- NIST AI Risk Management Framework: AI RMF Core. Documented evaluation, independent assessment and ongoing monitoring. Reviewed 2026-11-11.
Turn the finding into a decision
Research Desk: Commission a bounded oversight-effectiveness study with one decision type, agreed evidence access and independent adjudication. The deliverable should show transitions, unresolved cases, capacity constraints and one specific control change to test. DN sells research services; the engagement does not certify compliance or guarantee that a human-AI combination will outperform either participant.
Related DN research
Disclosure: Analysis and instrument results are editorial. Where labelled affiliate links appear, DN may receive compensation when you sign up or buy through them. Commercial relationships do not determine findings. No tested integration or guaranteed outcome is implied.
