Decentralised News · Agentic Finance · Decision research
By Heath Muchena · Source review 2026-11-11 · Instrument version 1.0.0
What Matters
A dependable reviewer should change its verdict when the evidence changes. Presentation tests ask what happens when the underlying work stays the same. DN’s matched test records six potential cues, an unchanged control and a fixed rubric. It helps teams identify an unstable review gate without pretending that every changed decision proves bias.
A confident answer can be wrong. A terse answer can be complete. An AI review system needs a way to distinguish those properties before its scores decide which work reaches a customer.
The commercial question is practical: would the same deliverable pass if its paragraphs were reordered, its author label removed or its explanation made more polished? If presentation changes approval while business correctness stays fixed, suppliers can optimise for the reviewer instead of the job. The appropriate response is a controlled comparison with adjudicated cases, not an assumption that longer, shorter or more assertive answers are inherently better.
A presentation test begins with a case whose relevant facts have already been checked. Save the original deliverable, the task, the permitted sources and the rule that determines a correct verdict. Have a person who understands the job confirm which transformations preserve its meaning.
This step sounds administrative but decides whether the experiment is interpretable. Removing a paragraph that contains the only warranty exclusion is a substantive change. Moving that paragraph is a presentation change. Replacing a precise figure with a vague description can alter the claim even if the editor considers the rewrite cosmetic.
Define the reviewer’s job narrowly. A factual checker should assess support and contradictions. A customer communications reviewer might legitimately assess tone and readability as well. A change that should leave one rubric unchanged may matter under another. Record the invariant being tested: for example, the same permitted-use rights, the same calculation and the same factual evidence.
Use cases from more than one difficulty level. Obvious successes establish a basic control, but borderline cases reveal whether a presentation cue can tip a consequential decision. Keep valid and defective cases in the set. A reviewer that becomes more permissive under polished writing can appear helpful on valid work while admitting more defects.
DN Evidence Block
Sources reviewed: 2026-11-11 · Evidence type: Primary research on model judges, supplemented by an original DN test protocol.
Scope: Presentation sensitivity in a bounded review task. Results from published benchmarks do not establish the behaviour of a buyer's current reviewer.
- The MT-Bench study describes position, verbosity and self-enhancement biases among the limitations of language-model judges. These are reasons to test a judge, rather than permanent performance estimates for current models.
- Justice or Prejudice? introduces CALM to evaluate multiple judge biases through controlled input modifications. Its approach supports separating presentation changes from substantive changes.
- ContextualJudgeBench examines whether judges follow contextual evaluation instructions. A reviewer can therefore need both a presentation test and a separate check that it applies the intended rubric.
Not established by this work: DN has not run the proposed matched tests against a live customer review system. Examples and calculator inputs are illustrative.
Original DN instrument · version 1.0.0
DN Presentation-Bias Test Set
Compare observed verdict-change rates for meaning-preserving variants and unchanged repeats. Enter counts from a completed, adjudicated test; the calculator does not run model evaluations.
Enter your assumptions or load a labelled example. Required blank numbers stay missing; unknown evidence remains unknown. Use nonpersonal labels only. This instrument makes no network requests and uses no persistent browser storage.
Use your own inputs
No current result. Complete inputs or load an illustrative example.
Meaning preservation and verdict correctness require independent adjudication. Different case mixes, dependent repeats and small samples limit comparisons. The tool calculates from submitted counts; it does not test a model or establish causation.
1. Candidate order
In a pairwise review, swap which answer appears first. Preserve candidate identities outside the prompt so that a verdict for “A” can be mapped back to the actual answer. Otherwise a label change can be mistaken for a preference change.
Practical fixture: Use a synthetic pair with one supported and one unsupported claim. Run A/B and B/A with the same evidence and map the verdict back to candidate identity. A switch in the printed letter alone is not a substantive reversal. Preserve the rationale so an adjudicator can check whether the reviewer followed the evidence or the position.
2. Length
Compare a concise complete explanation with a longer version containing the same decision-relevant claims. A shorter version that deletes evidence is not an equivalent candidate. Likewise, adding a correct new justification can genuinely improve the work.
Practical fixture: Create the longer version by adding explanation that neither introduces a new fact nor removes a condition. Have the equivalence check performed before scoring. Record token or word count as a presentation variable, not as proof of quality. An unchanged repeat remains necessary because two different verdicts can also reflect ordinary run-to-run variation.
3. Confidence language
Change unsupported certainty cues while holding the factual propositions constant. Test whether “clearly” or an emphatic conclusion changes the decision. Do not remove legitimate uncertainty from a claim whose uncertainty is part of its meaning.
Practical fixture: For a bounded fixture, keep the calculation and source unchanged while replacing an emphatic introduction with neutral wording. Do not change a probabilistic claim into a certain one; that would alter the substance. Check whether the same acceptance threshold and evidence requirement are applied in both versions.
4. Claimed authority
Present the same work with and without an author or model label. This tests reliance on identity cues. It does not establish that every identity field is irrelevant: a workflow that verifies a licensed signatory has a different task from blind content review.
Practical fixture: Use only fictional or properly authorized identity labels in the fixture. Compare anonymous review with the same content carrying a stated author role. Keep an independently verified authority requirement outside this test: content review and verification that a particular person may sign are different tasks, and should have different acceptance rules.
5. Formatting
Change paragraphs into a table, move a summary or adjust heading prominence without changing units, conditions or evidence. Check that the transformation has not made a crucial exception inaccessible.
Practical fixture: Preserve every row, unit, footnote and qualification when converting a paragraph into a table. Verify the rendered result before calling the pair equivalent. If a condition is omitted by the transformation, classify that as an input defect rather than blaming the reviewer for sensitivity to layout.
6. Surface fluency
Correct harmless grammar or polish the prose while preserving the business content. Where clarity is explicitly scored, analyse factual correctness and writing quality separately. Otherwise an appropriate style score can conceal an inappropriate correctness reversal.
Practical fixture: Use a business fact that remains unambiguous in both versions, then separate correctness from writing quality in the output. A polished version can legitimately earn a better style rating while leaving the eligibility or arithmetic verdict unchanged. The fixture asks whether those judgments remain separate under the stated rubric.
Methodology, limitations and corrections
DN constructed this protocol from published research on judge bias and contextual evaluation. The six cues are proposed test dimensions, not findings about a named current service. The illustrative arithmetic is synthetic. A real study should prerecord case selection, semantic equivalence rules, model configuration, randomisation, decision thresholds and treatment of unresolved cases. Report counts by cue and severity, keep an unchanged repeat control and preserve a separate held-out set for any revision. This article is about presentation sensitivity; overall judge accuracy, evidence independence and human override effectiveness need separate measurements.
Add an unchanged control and blind the reviewer
A model can give different answers to identical inputs. The test therefore needs an unchanged control: repeat the original case under the same recorded configuration without applying the presentation transformation. The control estimates observed instability in this sample. It does not turn a small experiment into a reliable population estimate.
| Record | Purpose |
|---|---|
| Original case and adjudicated result | Establish what should stay true |
| Transformed case and transformation note | Show what changed and what did not |
| Unchanged repeat | Expose ordinary run-to-run variation |
| Rubric and configuration | Keep the review task comparable |
| Normalised verdict and cited reason | Separate label flips from substantive reversals |
Run each review without showing the previous verdict or explaining which answer is the experimental variant. Randomise presentation order where appropriate. Retain the exact input, model version where available, settings, raw response and parsing rule. A different parser is another change to the system, not just a reporting detail.
For a scored rubric, define material change before inspecting results. A shift from 81 to 80 might be inconsequential; crossing an approval threshold might block a customer delivery. Record both the score change and the resulting action. This keeps the analysis focused on decisions rather than visually impressive fluctuations.
Use the DN Presentation-Bias Test Set
The DN instrument compares two descriptive rates: verdict changes on meaning-preserving variants and verdict changes on unchanged repeats. It reports their difference in percentage points. It does not identify a causal effect, establish statistical significance or decide whether a changed verdict was correct.
Consider an illustrative test with 40 transformed pairs. Eight change verdict after a presentation edit. An unchanged control contains 40 pairs and two change verdict. The observed rates are 20% and 5%, a difference of 15 percentage points. That is a reason to inspect the cases. It is not evidence that presentation causes a 15% error rate in production.
Adjudication supplies the missing direction. Was a previously wrong rejection corrected, or did a correct rejection become an approval? A flip count treats those outcomes alike. Preserve a separate transition record showing original correctness, variant correctness, action severity and reviewer rationale.
A single aggregate can also hide a concentrated problem. Inspect each cue and each task type before pooling them. Ten order-sensitive invoice cases require a different intervention from ten harmless style-score changes on marketing copy. Small subgroups should remain visible with their counts; percentages without denominators invite overinterpretation.
Repair the review process and repeat the matched cases
A useful finding leads to a specific change. If candidate order matters, review both orders or remove pairwise ordering from the acceptance step. If asserted authority dominates, strip irrelevant identity fields. If polished prose masks an arithmetic defect, require an independent calculation check. These are proposed controls whose effectiveness still needs testing.
Preserve a held-out set when changing the prompt or rubric. Rewriting instructions until the current examples pass can teach the reviewer the examples instead of the standard. Compare the revised reviewer on familiar regressions and unseen cases, including defects that the original system caught.
Decide in advance which failures require escalation. A reversible style disagreement may tolerate a different threshold from an unsupported financial number or an omitted contractual condition. A single “bias score” cannot capture those differences.
For procurement, ask vendors for the actual matched records, semantic equivalence checks and unchanged controls behind a presentation-robustness claim. A chart showing average agreement without case definitions leaves the important question unanswered: did the review decision follow the work, the packaging or ordinary instability?
For corrections, identify the claim, source and date when contacting Decentralised News. A new record-creation timestamp does not refresh its underlying evidence.
Change log: version1.0.0, 2026-11-11. Initial documentation review, original instrument and labelled examples.
Frequently asked questions
Does a changed verdict prove the reviewer is biased?
No. The original verdict may have been wrong, the transformation may have changed meaning, or repeated identical inputs may also vary. Adjudicate the transition and compare it with unchanged controls.
Should every review ignore writing style?
No. Style can be part of the customer’s requirement. Score style separately from factual correctness and decide which presentation changes should leave each part of the rubric unchanged.
Can we compare two providers with these rates?
Only with a matched case set, common transformation rules and a consistent configuration record. Different denominators, task mixes or parsing rules can make the comparison misleading.
Primary sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Published discussion of position, verbosity and self-enhancement biases in model judges. Reviewed 2026-11-11.
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. Controlled input modifications for studying judge biases. Reviewed 2026-11-11.
- ContextualJudgeBench. Evaluating adherence to contextual review criteria. Reviewed 2026-11-11.
Turn the finding into a decision
Research Desk: Commission a bounded review-gate assessment: one rubric, one decision type, a permitted case set and a documented presentation-variant protocol. The deliverable should include the adjudicated transitions, unresolved cases and a specific next test. Agree scope and access before commissioning; no favourable score or vendor endorsement is promised. DN sells research services, which is a commercial interest separate from the findings above.
Related DN research
Disclosure: Analysis and instrument results are editorial. Where labelled affiliate links appear, DN may receive compensation when you sign up or buy through them. Commercial relationships do not determine findings. No tested integration or guaranteed outcome is implied.
