Decentralised News · Editorial by Heath Muchena · Evidence reviewed 11 November 2026. Original DN analysis and local instrument; field testing is claimed only where expressly reported.
What Matters
An AI service can delete a document, stop retrieving it or refuse to discuss it while its trained model remains unchanged. A credible unlearning claim identifies the affected model, the data scope, the reference comparison and the remaining exposure. DN’s evidence record makes those boundaries visible without converting a checklist into a privacy certificate.
Decentralised News · Model Governance Research · Edition 1.0 · Reviewed 11 November 2026
DN Evidence Block
Established by sources: GDPR erasure provisions, EDPB guidance on model anonymity, and published unlearning methods and evaluation frameworks. Original DN work: a procurement decision framework and downloadable evidence record. Not measured: any provider’s deletion success, model memorization, extraction resistance, retained utility or legal compliance. All examples below are hypothetical.
The missing noun in “we deleted your data”
Ask what was deleted. A customer-facing message may refer to an uploaded file, a retrieval index, a conversation log, a fine-tuning dataset, an adapter or a deployed model. Those are different objects with different owners and retention paths. A deletion receipt becomes useful only when it identifies its object and scope.
Consider a hypothetical agent trained to answer questions about confidential supplier terms. Removing the current contract from its search index could stop one route to the answer. If the same terms were included in a training run, a model-weight claim needs different evidence. If a refusal rule merely blocks the supplier’s name, the visible behavior has changed without establishing what the model learned.
DN’s agent memory benchmark examines retrieval, memory privacy and deletion behavior. This article addresses the next question: what supports a claim that training influence has been removed from a particular trained model? Keeping those scopes separate prevents an ordinary storage deletion from being sold as a stronger technical result.
Four claims that require different evidence
| Claim | Evidence to request | What remains open |
|---|---|---|
| The source document was deleted. | Storage scope, deletion record, replicas and retention exceptions. | Copies, derived data and models previously trained on it. |
| The agent cannot retrieve the document. | Index removal and controlled retrieval tests across relevant stores. | Training influence and other ways of obtaining the information. |
| The service will not answer about the document. | Policy configuration and behavioral evaluation of the exposed interface. | Whether refusal is the only barrier and what other interfaces expose. |
| The trained model unlearned the target data. | Model identities, a specified unlearning method, suitable controls and multi-dimensional evaluation. | Residual risk outside the tested data, access model and assumptions. |
This is a scope table, not a ranking. Deleting retrieval data may be exactly the appropriate operation when it never entered training. The problem is claiming an effect on model weights that the evidence does not address. Before commissioning expensive model testing, establish whether the relevant training event occurred and whether the vendor can trace it.
A benchmark result is a bounded result
TOFU uses fictitious author information to create a controlled unlearning task. That design helps researchers know which facts were introduced and compare forgetting with retained performance. It does not turn a favorable result on fictitious profiles into proof that a service has removed a customer’s real documents from every training stage. [1]
MUSE evaluates several dimensions: verbatim and knowledge memorization, privacy leakage, retained utility, scale and successive requests. Its value for a buyer is the breadth of questions. Passing one response test would leave much of that evaluation unanswered. Published research results remain the authors’ results for the studied settings; DN has not rerun them. [2]
Ask vendors to identify which part of their claim a benchmark supports. A method might suppress target answers while damaging useful adjacent knowledge. It might preserve ordinary answers but still expose membership information to a more capable reviewer. It might perform adequately on one deletion request and deteriorate after many. These possibilities require separate evidence rather than an average “forgetting score.”
There is also a difference between an empirical comparison and a mathematical guarantee. Guo and colleagues study certified removal with a mechanism for linear classifiers. A citation to that work does not transfer its guarantee to an arbitrary hosted language model. The relevant question is whether the deployed method satisfies the theorem’s actual conditions. [3]
The reference model determines what “forgotten” means
A strong claim needs a counterfactual: what would an appropriate model look like had it not used the target data? A retained-data retraining reference can help address that question. Where such a reference is unavailable, the report should say so and explain the weaker comparison used. A before-and-after screenshot alone cannot supply the missing counterfactual.
Do not demand that every target-associated fact disappear. A fact may also be learned legitimately from retained material. Conversely, a target answer becoming unavailable does not establish that every trace of training influence has gone. The target must specify whether the request concerns particular records, private facts, verbatim passages, associations or a defined set of downstream behaviors.
For a hypothetical supplier contract, the forget set might include the original PDF, extracted text, a translated copy and synthetic question-answer examples generated from it. A test that deletes only the PDF’s exact byte sequence may address a much smaller scope than the commercial promise. The reviewer should record known duplicates and transformations without publishing the confidential material itself.
Separate held-out evaluation examples from examples used to tune the unlearning procedure. If every failed test is folded back into the method and the same tests are then reported as successful, the report describes adaptation to those tests. A useful final evaluation needs independent material and a declared procedure for handling further failures.
Unlearning can begin before a deletion request arrives
SISA research organizes training to limit individual data influence and reduce the work required for later removal. The practical procurement lesson is that deletion capability can depend on how training was designed. It is not necessarily a universal repair operation added to an already trained model. Its published speedups are task-specific research findings, not a saving DN applies to today’s language-model vendors. [4]
Before contributing proprietary training data, ask which model stages will receive it, how lineage is recorded and how deletion requests propagate. Ask whether the provider owns the base model or controls only an adapter. A supplier that controls one fine-tuning layer may be unable to alter an upstream model or a separately operated customer deployment.
The commercial issue is the service boundary. The contract, architecture diagram and evaluation report should describe the same system. “Our platform supports deletion” leaves too much unresolved when checkpoints have been exported, embedded into products or copied into recovery environments. Define the responsible party and the evidence each party must provide.
Technical evidence and erasure law answer different questions
Article 17 of the GDPR sets grounds for erasure and includes exceptions. Whether a particular controller must erase particular personal data requires an analysis of the processing and applicable provisions. A successful technical test does not make that legal analysis unnecessary, and a failed test does not by itself decide the legal remedy. This article is not a determination for an individual case. [5]
The EDPB’s Opinion 28/2024 also rejects treating every model trained on personal data as automatically anonymous. It calls for a case-specific assessment that considers extraction and information obtained through queries, among other evidence. Anonymity is a legal and factual assessment; “our chatbot refused this prompt” is a substantially narrower observation. [6]
Keep the technical report available to the qualified privacy reviewer, with its assumptions intact. Do not strip its limitations when translating it into a customer response. A concise response can identify what was removed, what was tested, which systems were outside scope and how unresolved concerns can be examined. That is more informative than an unsupported percentage.
The DN Model Unlearning Evidence Record
The instrument below attaches eight evidence areas to a named model and request. It records training lineage, model versions, a comparator, target exposure tests, privacy evaluation, retained utility, repeated requests and deployment replacement. “Evidence reference entered” means exactly that: the user supplied a reference. DN has not inspected the referenced artifact.
The output counts recorded references, unknown areas and explicit issues. Those counts describe the record’s contents. They are not a success rate, a privacy probability or an audit grade. A record with eight references may still contain weak tests. A record with fewer references may correctly disclose that the provider does not have access to the base model.
There is a deliberate scope check. If the chosen change affects only retrieval or output filtering, the result says it does not establish model-weight unlearning. When the scope is unknown, the record preserves that uncertainty. The tool never turns empty fields into proof or awards a compliance badge because every box is filled.
DN Model Unlearning Evidence Record
This record cannot certify unlearning, anonymity, legal compliance or a percentage of data forgotten. Selected evidence statuses are self-reported.
Use non-sensitive aliases and reference labels. Inputs remain in this page’s temporary memory; downloading creates a local JSON file. Nothing is submitted to DN.
No record has been built. Missing evidence will remain visible.
Use the record to change the next decision
Imagine two suppliers. Supplier A supplies a deletion receipt for an index and a refusal screenshot. Supplier B identifies the affected adapter, documents the target-data inventory, supplies a retained-data comparison and reports a utility regression. This hypothetical example does not rank real vendors. It shows why a disclosed limitation can be more decision-useful than an apparently perfect claim with an undefined scope.
The next action for A is to clarify whether training ever occurred and what its deletion receipt covers. The next action for B is to review whether the remaining exposure and utility tradeoff satisfy the specific request. Neither decision follows from counting references alone. The record makes missing questions explicit so that an engineering or privacy reviewer can address them.
Retain an immutable copy of each reviewed report and the model version it describes. Reopen the record when the model changes, an old checkpoint is restored or a new data source reintroduces the target material. DN’s recursive agent improvement watch covers the adjacent problem of systems changing their own prompts, memory and code; a deletion conclusion should not silently survive an unrelated deployment change.
Methodology, limitations and corrections
Edition 1.0, 11 November 2026. DN reviewed the linked legal texts, original research and benchmark documentation, then designed the evidence record. No model weights were downloaded, no training or unlearning was performed, and no customer data was probed. The framework is not a census of available methods or a finding that all current methods fail.
The exported JSON identifies schema dn.model-unlearning-evidence.v1, version 1.0.0, documentation review date, record creation time, user-selected input basis, references and limitations. Creation time is not the date a model was tested. Use non-sensitive aliases; the public tool is not a place to paste confidential forget sets. Machine-readable fields preserve uncertainty rather than manufacture a benchmark.
Change log: Initial edition separates retrieval deletion, filtering, empirical unlearning evaluation and formal guarantees. The article will require review if relevant law, authoritative guidance, method assumptions or documented deployment practices change. Corrections should identify the exact claim, version and supporting evidence; contact press@decentralised.news.
Frequently asked questions
Does removing a document from a vector database count as unlearning?
It can remove a retrieval route. Whether that resolves the actual request depends on the data’s history and required scope. It does not alone demonstrate a change to trained model weights.
Does a model refusing to answer prove the data is gone?
No. Refusal is an observable behavior of an interface. An unlearning claim needs evidence about the specified model and relevant exposure under a declared evaluation.
Can this tool certify GDPR compliance?
No. It organizes self-reported technical evidence for review. The applicable legal duties and conclusions require a separate assessment of the actual circumstances.
Why does DN not publish a forgetting percentage?
There is no single denominator in this review that would make such a percentage meaningful. Target recall, extraction risk, retained utility and deployment coverage answer different questions.
Primary sources
- TOFU: A Task of Fictitious Unlearning for LLMs — controlled benchmark design.
- MUSE — six-way language-model unlearning evaluation.
- Guo et al., Certified Data Removal from Machine Learning Models — formal guarantee in the studied setting.
- Bourtoule et al., Machine Unlearning — SISA training architecture.
- GDPR — Article 17 and its conditions.
- EDPB Opinion 28/2024 — model anonymity and personal-data processing.
