Decentralised News · Editorial by Heath Muchena · Evidence reviewed 11 November 2026. Original DN analysis and local instrument; field testing is claimed only where expressly reported.
What Matters
A text watermark’s survival rate is meaningful only for a specified generator, detector, threshold, text population and transformation. Count missed detections and false positives alongside successful detections, and use matched examples to measure retention after editing. DN’s instrument summarizes declared test counts; it does not inspect text, identify authors or present an illustrative scenario as a real watermark benchmark.
Decentralised News · Media Evidence Research · Edition 1.0 · Reviewed 11 November 2026
DN Evidence Block
Source-backed: original watermarking research, official SynthID implementation guidance, a dated Anthropic method announcement and NIST interval mathematics. Original DN work: the evaluation design, paired-retention calculation and local record. Not measured: any provider’s watermark robustness, false-positive rate, text quality or detection availability. No text was submitted to a detector.
“It survived editing” leaves the important questions unanswered
A publisher says its AI text watermark survives formatting changes. Another demonstrates detection after a summary. A third reports results on long passages while customers mostly publish short alerts. Those claims concern different experiments. They should not be collapsed into a single market-wide percentage.
For an editorial team, the practical question is narrower: does a particular provenance signal remain useful through the transformations the team actually performs, at an acceptable false-positive rate? The answer requires a defined test population, positive examples with known provenance, negative controls and a detector configuration fixed before the final evaluation.
DN’s content-platform comparison evaluates production workflows. This article examines evidence about a specific statistical signal in generated text. It does not rank writing platforms, identify individual authors or treat a detector result as a finding of misconduct.
A watermark is not the same as a generic AI-writing guess
The original language-model watermark research describes embedding a statistical signal during generation and then testing for that signal. Detection is tied to a particular mechanism. This differs from guessing that a passage “sounds like AI” because of its vocabulary or style. A detector for one target watermark cannot establish that all other text is human-written. [4]
SynthID-Text is another generation-time scheme, described and evaluated by its authors in Nature. Its published findings support conclusions about the configurations and experiments in that paper. They are not a DN measurement of every model, language, editing workflow or future deployment bearing the SynthID name. [1]
Google’s toolkit documentation describes watermarking configuration and detection integration. The official research repository explicitly says it is not intended for production use and supplies particular detector implementations. A downloaded research notebook should therefore be recorded by version and evaluated within its scope, not presented as a universal production validator. [3] [2]
Anthropic’s August 2026 announcement, updated in September, describes its chosen SynthID-Text-based approach, limitations for short or constrained outputs and a restricted detector-access program. Those are vendor statements at the cited date. DN has not tested that implementation or established access for a particular reader. [6]
Define the experiment before choosing the headline
| Required context | What to record | Why it changes interpretation |
|---|---|---|
| Generator and watermark | Model version, watermark scheme, configuration and non-secret key alias. | Not every model output necessarily contains the target signal. |
| Detector | Implementation, version, tokenizer and applicable configuration. | Different detectors can produce different statistics. |
| Decision threshold | The fixed rule and the separate material used to choose it. | Tuning on the final test set biases the reported result. |
| Text population | Domain, language, lengths, sampling and exclusions. | A long essay result need not apply to a short market alert. |
| Transformation | The exact editorial operation and software version. | “Edited” can describe very different changes. |
| Ground truth and controls | Permitted generation records and known non-target-watermarked examples. | The confusion counts are meaningless without reliable labels. |
Keep secret keys and private source text outside the public record. A key alias can connect a review to a restricted evaluation environment without disclosing the key itself. Record how the evaluator knows which examples carry the target watermark; do not use the detector’s own output as ground truth.
Negative controls mean examples not watermarked with the target mechanism or key. They need not all be human-written. They can include appropriate material from other generators, provided the labels and purpose are explicit. This prevents “not detected by this tool” from quietly becoming “proved human.”
Count the four outcomes
For the specified transformed test corpus, a true positive is a known target-watermarked example detected by the configured test. A false negative is a target example that is missed. A false positive is a non-target control that is detected. A true negative is a control that is not detected.
| Metric | Formula | Question answered |
|---|---|---|
| Detection rate | TP / (TP + FN) | What share of the known target examples was detected? |
| False-positive rate | FP / (FP + TN) | What share of the non-target controls was incorrectly detected? |
| Precision in the entered sample | TP / (TP + FP) | Among this sample’s detections, what share had the target ground truth? |
These formulas describe an entered test. They do not establish a population rate without a defensible sampling design. When no examples are detected, sample precision is undefined. Reporting it as zero or 100% would conceal the missing denominator. The tool preserves a null value in that case.
A robust evaluation also keeps failures and exclusions visible. If the detector rejects unsupported scripts, short inputs or corrupted documents, report those separately rather than silently removing them from the denominator. The current DN calculator assumes each entered example received a binary result; unresolved cases require a separate count and cannot be smuggled into true negatives.
Survival requires pairing, not just two averages
Suppose a wholly illustrative test has 100 known target-watermarked examples. The detector finds 90 before an editorial transformation and 60 afterward. Dividing 60 by 90 does not reveal how many original detections survived. Some previously missed examples might be detected afterward.
To measure retention, track the same examples with the same detector, key and fixed threshold. In this illustrative paired table, 55 examples are detected both before and after:
| Same positive cohort | Detected after | Not detected after |
|---|---|---|
| Detected before | 55 | 35 |
| Not detected before | 5 | 5 |
Post-transformation detection is 60/100, or 60%. The change from baseline is minus 30 percentage points. Retention among baseline detections is 55/90, approximately 61.11%. These three numbers answer different questions. DN’s tool requires the intersection count before calculating paired retention.
The tool also rejects impossible paired counts. An intersection cannot exceed either marginal total, and it must be large enough for the stated cohort size. This arithmetic check cannot authenticate the test, but it prevents a common inconsistency from becoming a polished exported result.
False positives change the meaning of a high-looking precision
The tool’s illustrative example adds 100 non-target controls, with two false positives and 98 true negatives. Detection is 60%, false-positive rate is 2%, and precision in that balanced sample is 60/62, approximately 96.77%. These are calculated scenario values, not observations about a named watermark.
Now consider a separate hypothetical deployment of 100,000 items in which only 1% carry the target watermark. If, solely for illustration, the same 60% detection and 2% false-positive rates applied, there would be 600 true positives and 1,980 false positives. Precision would be about 23.26%, not 96.77%.
The arithmetic shows why a balanced evaluation sample cannot establish how likely a real-world detection is to be correct. The prevalence of the target signal and the representativeness of both classes matter. Do not present a detector statistic or a test-set precision as an individual author’s probability of having used AI.
Text length and transformations belong in the result
Research on watermark reliability studies how evidence changes under transformations and as more tokens are available. Its findings are conditional on the schemes and evaluation procedures studied. They do not support a universal statement that all watermarks survive all paraphrasing, or that every edit makes detection impossible. [5]
For a newsroom, choose a small set of ordinary, authorized workflow transformations: the actual formatting step, the approved shortening process or the translation path used in production. Run each as a separately identified condition. A summary and a format conversion should not share a single “edited text” row.
Match token-length reporting to the detector’s tokenizer and retain the length distribution rather than only the mean. If a transformation substantially shortens text, distinguish the changed amount of available evidence from the transformation’s other effects. Keep mixed-source documents and excerpts as their own cohorts instead of assuming whole-document labels apply to every passage.
Zero false positives is not zero uncertainty
A sample of 100 controls with no false positives does not establish a population false-positive rate of zero. Under an asserted independent Bernoulli sampling model, a two-sided 95% Wilson interval has an upper bound of approximately 3.70% for that example. The interval formula is documented in the NIST handbook. [7]
The tool offers conditional Wilson intervals only when the user explicitly asserts the relevant independent, representative trial assumptions. Otherwise it reports the raw rates without intervals. Neither choice fixes a biased corpus, repeated threshold tuning, near-duplicate examples or a detector trained on the test material.
For comparisons between methods, preserve paired outcomes and use an appropriate statistical design rather than declaring a winner because two rounded percentages differ. This instrument does not test statistical significance between detectors or claim that its optional intervals establish general robustness.
The DN Watermark Test Record
The instrument records the test context and confusion counts, calculates the three rates and optionally adds paired retention. It can export a stable JSON record containing the method assumptions, source references and unresolved limits. The default fields are blank; a separate button loads the explicitly illustrative scenario.
No text is uploaded or classified. Entering a generator’s name does not connect to that provider. The tool does not obtain a detector key, call a private API or decide whether a person used AI. It is a record and arithmetic layer for evaluations conducted elsewhere, with user-reported evidence kept separate from DN’s documentation review.
DN Watermark Test Record
This calculator does not inspect text or run a watermark detector. It summarizes user-reported or illustrative test counts. It cannot establish human authorship, plagiarism, truth or misconduct.
Use non-sensitive descriptions. This tool makes no external requests and retains inputs only in temporary page memory. Downloading creates a local JSON file.
No calculation yet. Blank values are not zero.
Methodology, limitations and change log
Edition 1.0, 11 November 2026. DN reviewed original watermarking papers, official implementation documentation, a dated provider statement and NIST interval mathematics. The examples are authored scenarios. No watermark generator, detector, translation pipeline or adversarial editing experiment was run.
Exports use dn.watermark-test-record.v1, version 1.0.0. They distinguish local record creation from test execution, preserve the input basis and use null for undefined metrics. Positive and negative cohorts are both required. Optional paired comparisons require the same positive examples and fixed configuration; the tool checks count consistency but cannot verify that assertion.
Change log: Initial edition replaces an unsupported vendor survival ranking with a reproducible evaluation record. Revisit the source review when implementations or documented access change. Report corrections with the affected formula, configuration and evidence to press@decentralised.news.
Frequently asked questions
Does a negative result prove that a human wrote the text?
No. It means the configured test did not detect its target signal under the tested conditions.
Can the tool calculate survival from before-and-after detection totals?
It can calculate the change in detection rate for a matched cohort. Retention additionally requires the count detected in both conditions.
Are the illustrative numbers SynthID or Claude results?
No. They are synthetic counts chosen to explain the arithmetic and cannot be attributed to any provider.
Can a watermark establish that a financial statement is true?
No. A provenance signal and a factual claim answer different questions. Financial evidence requires separate verification.
Primary sources
- SynthID-Text research, Nature 2024.
- Official SynthID Text research implementation.
- Google responsible AI toolkit: SynthID.
- A Watermark for Large Language Models.
- On the Reliability of Watermarks for Large Language Models.
- Anthropic’s dated text-watermark explanation.
- NIST confidence intervals for proportions.
