Decentralised News · Agentic Finance · Decision research
By Heath Muchena · Source review 2026-11-11 · Instrument version 1.0.0
What Matters
Five matching verdicts do not tell a buyer how many independent checks occurred. A panel may share sources, preprocessing errors or persuasive reasoning. DN’s Source-Independence Test Pack separates those dependencies from voting arithmetic. The useful purchase question is whether an additional reviewer catches a material error that the existing process misses.
A panel of reviewers creates an appealing picture: several systems inspect the same work, disagree where necessary and converge on a dependable result. That picture leaves out how each reviewer obtained its evidence. Five votes produced from one incomplete summary can reproduce one mistake five times.
The issue is not whether panels can help. It is what a buyer is purchasing when paying for them. More calls, more model brands and a more elaborate debate are observable inputs. An independently verified improvement in the final decision is the outcome that matters.
DN Evidence Block
Sources reviewed: 2026-11-11 · Evidence type: Primary research on correlated model-judge errors and multi-agent debate, plus an original DN measurement protocol.
Scope: Review panels applied to a fixed, adjudicated decision task. Published experiments do not establish a general failure rate for current commercial agents.
- Nine Judges, Two Effective Votes, a May 2026 preprint, reports substantial error dependence among the judges and tasks it tested. Its effective-vote estimate is not a universal conversion from models to independent reviewers.
- Talk Isn’t Always Cheap documents correct-to-incorrect transitions during debate in its experiments. Agreement after discussion can therefore differ from independent confirmation.
- Debate or Vote separates voting effects from debate effects. Its findings motivate a voting-only baseline rather than assuming discussion adds value.
Not established by this work: DN has not benchmarked a live reviewer panel. The calculator contrasts mathematical scenarios under stated assumptions.
Original DN instrument · version 1.0.0
DN Source-Independence Test Pack
Illustrate majority voting under two hypothetical dependence assumptions. This is a mathematical demonstrator, not a measurement of real reviewer independence.
Enter your assumptions or load a labelled example. Required blank numbers stay missing; unknown evidence remains unknown. Use nonpersonal labels only. This instrument makes no network requests and uses no persistent browser storage.
Use your own inputs
No current result. Complete inputs or load an illustrative example.
The calculation omits unequal reviewer quality, correlated but nonidentical errors, ambiguous labels, abstention, cost and severity. It cannot infer real accuracy, independence or safety.
1. Every reviewer reads the same incomplete source
An agent panel asked whether a plan permits commercial reuse may all inspect the same pricing summary. If the decisive restriction lives in the licence, unanimous approval verifies agreement with the summary rather than the entitlement. Record which source supports each material claim, its version and which reviewer actually inspected it.
Source diversity should be relevant. Five websites repeating a supplier announcement do not provide five independent records of the underlying terms. A source map should connect secondary accounts to their origin. It should also distinguish the vendor’s statement from an independently observed delivery or an executed contract.
2. Preprocessing removes the same condition for everyone
Reviewers may receive different prompts but the same extracted table, cached answer or generated summary. A missing footnote, truncated row or dropped negative sign then sits upstream of every vote. The panel cannot recover evidence it never sees unless the workflow explicitly permits retrieval beyond that packet.
Preserve both the original material and the exact representation supplied to each judge. For a narrow pilot, introduce a known condition into a permitted test document and verify whether it survives extraction. This is a proposed diagnostic, not evidence that a named parser is faulty.
3. Later reviewers can see the earlier verdict
A reviewer asked to assess a confident prior explanation is doing a different job from one asked to judge the original evidence alone. Preserve initial blind verdicts before any discussion. Record every change after communication, including whether that change improves or worsens the decision.
4. Model diversity is treated as measured independence
Different provider names, architectures or prompts may be useful experimental variations. They do not, by themselves, demonstrate uncorrelated mistakes on the buyer’s actual cases. Measure overlap in errors against adjudicated outcomes. Also report the number and type of cases; a small sample can make unrelated systems look identical or different by chance.
5. The debate process rewards agreement
A workflow might continue until the panel reaches consensus or ask a chair to reconcile every objection. That process can produce a clean final answer while concealing unresolved evidence. The relevant endpoint is a justified decision, including an explicit unresolved state when the evidence does not support closure.
Do not diagnose the cause from a shared error alone. Common training information, task difficulty, a shared source and direct persuasion are different explanations. The proposed experiment varies evidence access and communication separately so that the operator can investigate which dependency matters.
6. A valid minority objection disappears from the final answer
Even an independently useful reviewer can add no protection if the final aggregator omits its condition. Save dissent as a structured record: the disputed claim, the contrary evidence, its materiality and the reason the final decision accepted or rejected it.
A panel can be assessed through a transition table rather than a single agreement percentage. Compare the initial independent decision with the decision after discussion and aggregation. The correct answer must come from an adjudication process outside the panel being tested.
| Initial result | Final result | Interpretation |
|---|---|---|
| Wrong | Correct | Useful correction to investigate and reproduce |
| Correct | Wrong | Harm introduced by the later process |
| Wrong | Wrong | Persistent error; agreement adds no evidence |
| Correct | Correct | Preserved success, with incremental cost still to assess |
| Unresolved | Any result | Keep separate until there is sufficient evidence to adjudicate |
Record severity as well as counts. Missing a purchase-changing licence restriction and disagreeing over a harmless wording preference should not cancel each other inside an average. A minority flag deserves examination because of its evidence, not simply because it is a minority position.
Methodology, limitations and corrections
DN synthesised the cited primary studies into a source-independence protocol. The six mechanisms describe possible failure paths and proposed checks; they are not a frequency ranking. Published task-specific observations remain separate from the synthetic binomial example. A real panel study should document adjudication, source access, initial and final verdicts, case selection, held-out evaluation, unresolved cases and cost. The tool is version 1.0.0 and performs only the arithmetic described above.
Use the DN Source-Independence Test Pack
The interactive instrument illustrates why independence matters. It compares majority-vote error under two deliberately simplified assumptions: identical independent error probabilities and perfectly shared errors. It does not estimate the dependence of a real panel.
With five reviewers and an assumed individual error probability of 20%, an independent majority is wrong when three, four or five reviewers are wrong. The binomial calculation gives 5.792%. If every reviewer makes the same errors on the same cases, majority error remains 20%. Both numbers come from the assumptions entered into the model, not from a DN benchmark.
The real protocol needs a richer record. Assign each case a stable ID and an adjudicated result. Save the original evidence, the packet supplied to each reviewer, source provenance, reviewer configuration, initial verdict, post-discussion verdict and final action. Separate cases where evidence is genuinely missing from cases where a reviewer overlooked available evidence.
Compare at least three bounded routes: the current single-review process, independent reviews aggregated without discussion, and the proposed panel with communication. Hold the business task and available evidence constant where the comparison requires it. Then repeat with deliberately separated evidence sources. An additional reviewer earns its place through material corrected decisions, not through a larger vote count.
Buy a demonstrated improvement in one decision
Start with a recurring decision whose correct resolution can be established: a calculation, a document condition or a permitted action under a supplied policy. Define the cases that require human adjudication and the cases the panel must leave unresolved. Avoid using the panel’s own consensus as its test label.
Set the improvement criterion before tuning. It might be fewer material missed defects with no unacceptable rise in false rejection, measured at a stated review cost. Preserve a held-out case set and record the version of every changed component. Otherwise a panel can be repeatedly adjusted to the examples used to sell it.
Account for the complete review process. Source retrieval, adjudication, escalations and a final human decision can cost more than the model calls. A panel that catches an additional serious error may justify that cost. A panel that reproduces the same result with more convincing prose has a different commercial value.
In a procurement review, ask for initial independent verdicts, source maps and case-level transitions. If the supplier can provide only a final consensus score, the independence claim remains unverified. That is a specific evidence gap the next pilot can resolve.
For corrections, identify the claim, source and date when contacting Decentralised News. A new record-creation timestamp does not refresh its underlying evidence.
Change log: version1.0.0, 2026-11-11. Initial documentation review, original instrument and labelled examples.
Frequently asked questions
Does using different model providers establish independence?
No. It is an experimental variation worth recording. Independence must be investigated through the errors and evidence paths of the specific decision task.
Should a minority reviewer always override the majority?
No. Inspect the objection and its supporting evidence. The minority can be right, wrong or unable to resolve the case; vote position alone does not decide.
Is debate worse than voting?
The answer depends on the task and configuration. Compare both against adjudicated outcomes and record correct-to-wrong as well as wrong-to-correct transitions.
Primary sources
- Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels. Task-specific correlated error findings in a May 2026 preprint. Reviewed 2026-11-11.
- Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate. Correct-to-incorrect transitions during model debate. Reviewed 2026-11-11.
- Debate or Vote. Separating majority-voting and debate contributions. Reviewed 2026-11-11.
Turn the finding into a decision
Research Desk: Commission one bounded reviewer-panel assessment with a fixed decision, a permitted evidence set and independent adjudication. The deliverable is a source-dependency map, case-level transition record and recommendation for the next test. Agree scope before commissioning. DN sells research services; a paid assessment does not promise a favourable verdict or certification.
Related DN research
Disclosure: Analysis and instrument results are editorial. Where labelled affiliate links appear, DN may receive compensation when you sign up or buy through them. Commercial relationships do not determine findings. No tested integration or guaranteed outcome is implied.
