Decentralised News · Agentic Finance · Decision research
By Heath Muchena · Source review 2026-11-11 · Instrument version 1.0.0
What Matters
A reviewer that catches defects can still make a workflow worse by rejecting valid work. Count those decisions separately, follow the work through repair and preserve unresolved cases. DN’s Reviewer Confusion Matrix shows how an apparently respectable accuracy score can conceal an expensive rejection queue. It provides arithmetic, not a verdict on any vendor.
A rejected deliverable creates work. Someone must inspect the objection, gather evidence, change the output or persuade the system that no change is needed. If the deliverable was already correct, that labour buys no improvement. A hurried repair can even introduce the defect the original version avoided.
False rejection deserves its own procurement question: how often does the reviewer block valid work, and what happens next? The answer requires adjudicated cases and final outcomes. Counting warnings, edits or escalations alone cannot establish that the reviewer helped.
DN Evidence Block
Sources reviewed: 2026-11-11 · Evidence type: Primary research on correctness-based judging and self-correction, together with official agent-evaluation guidance.
Scope: A bounded binary acceptance gate with independently adjudicated work. Unresolved cases remain separate from the confusion matrix.
- JudgeBench builds evaluations around objectively assessable response correctness, highlighting the distinction between a preferred answer and a correct one.
- Confidence v.s. Critique separates preserving correct responses from correcting wrong ones. The DN protocol applies that transition logic to the output of a review-and-repair workflow.
- Anthropic’s agent-evaluation guidance discusses brittle exact-match grading and calibration of model judges with human judgement. It does not validate this proposed DN tool.
Not established by this work: The eight mechanisms are proposed diagnostic categories. DN has not measured their frequency in a live customer workflow.
Original DN instrument · version 1.0.0
DN Reviewer Confusion Matrix
Measure a completed, adjudicated review sample. Enter valid and defective cases with binary decisions; put cases lacking a usable verdict or ground truth in the separate unresolved count.
Enter your assumptions or load a labelled example. Required blank numbers stay missing; unknown evidence remains unknown. Use nonpersonal labels only. This instrument makes no network requests and uses no persistent browser storage.
Use your own inputs
No current result. Complete inputs or load an illustrative example.
Ground truth, case selection and handling cost must be established outside the calculator. Binary decisions omit partial acceptance and severity; report those separately. No production accuracy or cost saving is inferred.
1. A valid format is treated as invalid
The task permits a table or a paragraph, but the reviewer expects one template. Establish which fields and relationships are required, then distinguish format preference from a missing business requirement. Some formats are mandatory; the defect lies in rejecting an allowed alternative.
Proposed test: Prepare two permitted representations containing the same required fields, such as a table and a paragraph. Keep a separate defective variant with one required field missing. The reviewer should recognize the material omission without treating the allowed format change as the defect.
2. A correct answer uses different words
A reference answer can be useful without being the only acceptable wording. If the gate compares phrasing too narrowly, a valid equivalent may fail. Preserve the exact rejected claim and the reason an adjudicator considers it equivalent. Do not assume synonyms preserve technical or contractual meaning without checking.
Proposed test: Use a pair whose equivalence has been checked by someone familiar with the subject. Preserve quantities, units, conditions and exceptions. A superficial paraphrase can accidentally change a right or an obligation; the equivalence label must be justified before using the pair to judge the reviewer.
3. A legitimate exception is mistaken for a rule violation
A standard process may permit an authorised exception, but the reviewer sees only the default rule. The evidence packet should identify the applicable exception, its scope and its authority. An undocumented exception remains unresolved; a reviewer is not obliged to infer permission that has not been supplied.
These categories matter because the fix differs. A formatting repair, a semantic check and an exception-authority record solve different problems. Grouping all three as “the model is too strict” makes the next prompt change difficult to evaluate.
Proposed test: Provide the default rule and a scoped, authorized exception in the same synthetic packet. Check whether the reviewer identifies which rule governs the case. Then remove the authorization record and verify that the output becomes unresolved rather than assuming permission or claiming a demonstrated violation.
4. Missing evidence is reported as proof of error
A reviewer may be unable to confirm a claim because a document is inaccessible or the case is incomplete. That supports an unresolved result or a request for evidence. It does not establish that the work is wrong. Record which evidence was missing and whether the supplier had an obligation to provide it.
Proposed test: Withhold one document that is necessary to establish the claim while leaving the deliverable unchanged. The desired result depends on the declared gate, but the explanation must distinguish missing evidence from a contradiction. Record the request for evidence and the person who can resolve it.
5. A style preference overrides the acceptance rule
The reviewer asks for a longer explanation, a different tone or extra context when the contract only requires a correct calculation. If style matters, make it a separate explicit criterion. Otherwise unnecessary rewriting can consume the savings the automation was supposed to create.
Proposed test: State a correctness-only acceptance rule, then vary harmless tone or length. Run a separate rubric that explicitly scores style. This helps distinguish a misplaced style objection from a legitimate response to the second task. Do not combine the two verdicts into one unexplained quality label.
6. An unresolved case is forced into rejection
Some systems provide only pass or fail even when the underlying question cannot be adjudicated. That can be a deliberate operational policy, but its cost should not be labelled a confirmed defect. Keep the distinction between “blocked pending evidence” and “shown to be invalid” in the record.
A useful review output contains the disputed requirement, the relevant evidence and a proposed resolution. A confident demand to improve the answer without those details is difficult to act on and difficult to audit.
Proposed test: Include a case that the reference adjudicators cannot resolve from the available packet. Test whether the workflow preserves the reason for blocking or escalation. A policy may prohibit proceeding while evidence is missing, but that operational block is still different from proving the submitted work false.
7. An old rule remains active after the requirement changes
A reviewer can enforce a retired template or superseded policy correctly while making the wrong business decision. Tie each case to the rule version and effective date. Review history should reveal whether the work or the gate used the outdated requirement.
Proposed test: Attach the effective dates of the old and new rule to a synthetic case. Preserve the event date that determines applicability. The reviewer should identify the governing version; choosing whichever document appears last in the packet is not an adequate authority rule.
8. The requested repair damages valid work
A customer changes a correct figure, deletes a necessary qualification or adds unsupported certainty to satisfy the reviewer. A final acceptance flag then conceals a worse deliverable. Preserve the original and repaired versions and adjudicate both independently.
| Original work | After repair | What to count |
|---|---|---|
| Valid | Valid | Unnecessary rework if no required improvement occurred |
| Valid | Defective | Repair-induced defect |
| Defective | Valid | Successful correction |
| Defective | Defective | Unresolved defect plus repair cost |
The review and repair stages need separate owners in the evaluation record. A reviewer may identify a real issue while the repair agent implements the wrong change. Conversely, a repair agent may rescue an imprecise review. The complete workflow is the customer’s experience, but those different mechanisms require different remedies.
Proposed test: Save a valid original, the requested change and the repaired version. Ask an independent adjudicator whether the repair preserved every material condition. A final pass flag cannot erase a newly wrong figure or omitted limitation. Count both the unnecessary intervention and any resulting defect.
Methodology, limitations and corrections
DN derived the eight diagnostic categories from the practical structure of a review gate and the cited evaluation research. They are a proposed taxonomy rather than measured prevalence. The confusion matrix and labour example are synthetic and reproducible from the displayed inputs. A real study should preserve the original requirement, case version, initial output, reviewer objection, adjudication, repair and final delivered state. Unresolved cases stay visible, and any policy change should be tested on held-out valid and defective cases.
Use the DN Reviewer Confusion Matrix
Consider 100 synthetic cases with a completed binary decision and independently established correctness. Ninety are valid and ten are defective. The reviewer catches eight defects but rejects eighteen valid cases. It correctly accepts seventy-two valid cases and misses two defects.
Overall accuracy is 80%. Yet 20% of valid work is rejected, and only eight of the twenty-six rejections identify a genuine defect: rejection precision is about 30.8%. A team looking only at accuracy could miss the burden created for its delivery staff.
The calculator keeps the denominators explicit. False-rejection rate uses valid cases. Defect detection uses defective cases. Rejection precision uses all rejected cases. Cases without an adjudicated truth or a completed binary decision are counted separately as unresolved and excluded from these rates.
Enter the average handling time and loaded hourly cost only if those inputs have a defensible basis. Eighteen unnecessary rejections taking six minutes each at 40 currency units per hour create 72 units of handling cost in this illustrative sample. That excludes delay, customer effort and any new defect introduced during repair. Use one currency and do not present the result as total business loss.
Improve the gate without making it indiscriminate
The objective is not to minimise rejection at any price. Accepting every deliverable eliminates false rejection while missing every real defect. Compare both kinds of error and define unacceptable failure severity before tuning the gate.
Build matched pairs where possible: one valid case and one minimally defective variant. A useful reviewer should distinguish the material change without being distracted by harmless presentation differences. Keep some pairs out of the tuning process and test the revised gate on them.
Route uncertainty according to the decision. A reversible internal draft may wait for clarification. A consequential external action may need to remain blocked until evidence arrives. Either route should record the reason and cost rather than quietly converting uncertainty into a factual defect claim.
Finally, follow sampled cases through delivery. A gate that reduces its warning count but increases missed defects is not improved. A gate that preserves defect detection while eliminating unsupported objections has a concrete benefit: less work is spent repairing outputs that already met the requirement.
For corrections, identify the claim, source and date when contacting Decentralised News. A new record-creation timestamp does not refresh its underlying evidence.
Change log: version1.0.0, 2026-11-11. Initial documentation review, original instrument and labelled examples.
Frequently asked questions
Is a high rejection rate evidence of a careful reviewer?
Not by itself. Rejections can be correct, incorrect or unresolved. Measure rejection precision and missed defects using independently adjudicated cases.
Should missing evidence count as false rejection?
It depends on the agreed task. If the deliverable was required to contain that evidence, its absence may be a defect. If the evaluator simply cannot establish the truth, preserve an unresolved category.
Why follow the repair after the reviewer passes it?
The repair may introduce a new defect or remove a necessary condition. A passing flag is not a substitute for checking the delivered outcome.
Primary sources
- JudgeBench: A Benchmark for Evaluating LLM-based Judges. Correctness-grounded judge evaluation. Reviewed 2026-11-11.
- Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs. Separating preservation of correct outputs and correction of wrong outputs. Reviewed 2026-11-11.
- Demystifying evals for AI agents. Limits of brittle grading and need for judge calibration. Reviewed 2026-11-11.
Turn the finding into a decision
Research Desk: Commission a review-queue assessment for one deliverable type. Define a permitted case sample, independent adjudication and a repair-tracking window. The output is a confusion matrix, evidence gaps and a costed next test. DN sells research services; commissioning an assessment does not establish that a reviewer is suitable for a regulated or consequential decision.
Related DN research
Disclosure: Analysis and instrument results are editorial. Where labelled affiliate links appear, DN may receive compensation when you sign up or buy through them. Commercial relationships do not determine findings. No tested integration or guaranteed outcome is implied.
