Schreiben Sie nicht mehr [MANDANT]: So anonymisieren Sie ein Dokument, bevor Sie es einer KI geben

Three redactions that were not

In January 2019, lawyers for Paul Manafort filed a response to the special counsel's claim that their client had lied to prosecutors. Parts of it were blacked out. Reporters copied the blacked-out passages and pasted them into a new document, where they read perfectly well. The hidden text described Manafort sharing 2016 polling data with Konstantin Kilimnik. The black bars were a visual layer sitting on top of the text. Nobody had removed the text underneath.

In September 2023, the Federal Trade Commission's case against Microsoft's acquisition of Activision Blizzard, in the Northern District of California, produced the same failure with more money attached. Microsoft supplied the court with a link to its admitted exhibits on 14 September, and the court uploaded them to the public case page. The exhibits had not been redacted. Within a week, Microsoft's internal plans for future consoles and games were in the press. When people assumed the court had leaked them, the FTC's public affairs director wrote: "The FTC was not responsible for uploading Microsoft's plans for its games and consoles to the court website." Judge Jacqueline Scott Corley's order said the same thing in plainer terms. The parties had been ordered to provide redacted exhibits, and Microsoft provided the link.

In April 2025, in the first days of the FTC's antitrust trial against Meta in the U.S. District Court for the District of Columbia, Meta's slides carried redactions that could be removed by copying the text out of the PDF. Underneath sat material from other companies, including a line that iPhone users tend to prefer Apple's own Messages app, and a Snap slide on competitors, TikTok among them, that were thriving. An Apple attorney called the disclosures "egregious". Snap's attorney described a "cavalier approach and casual disregard" for the companies dragged into the case. Meta's own attorney, Mark Hansen, suggested that a third party outside the trial team handle redactions from then on.

Three cases over six years, with experienced counsel on every side, and none of them was a failure of redaction technique. Each was a failure to check what the file contained before it left the building. A black bar is a drawing. Deletion is a different operation, and you confirm it happened by trying to undo it.

Four words that are not synonyms

Lawyers use redaction, anonymization, pseudonymization and tokenization as if they meant the same thing. Regulators do not, and the difference decides whether you still hold personal data when you are done.

Article 4(5) of the EU General Data Protection Regulation is worth reading once. Pseudonymisation is processing personal data "in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately". The definition assumes a mapping exists. The mapping is the point. Most products that sell an "anonymization" feature are doing pseudonymization, because the lawyer needs to know who [LENDER-1] was when the answer comes back.

That is not a criticism. Pseudonymization is the right tool for feeding a document to a model, for the reason the next section gets to. It does mean the document is still personal data, and still carries whatever confidentiality and data residency duties applied before you touched it.

Your placeholder changes the answer

The choice of placeholder is not cosmetic. It changes what the model can reason about, and a paper published this month measured by how much.

In September 2026, Deußer and colleagues published "On the Impact of Anonymization on the Performance of Large Language Models" (arXiv 2609.11335, first version 10 September, revised 15 September). They ran five language models across eleven benchmarks on original and anonymized inputs. Then they compared five anonymization techniques on GPT-4o mini across three of those benchmarks: redaction (the entity disappears), generalization (London becomes [CITY]), pseudonymization (London becomes a stable, entity-unique label such as LOC-1), randomization (a fixed random string) and masking (the first letters kept, the rest starred out).

The line that matters, quoted in full:

Generalization (e.g., "London" → "[CITY]") and Redaction (e.g., "London" → "") consistently resulted in the largest performance drops.

On MuSR, a multi-step reasoning benchmark, generalization scored 0.58 and redaction 0.52, against 0.70 for pseudonymization and 0.73 for the untouched input. The authors' conclusion: techniques "preserving unique identifiers for distinct entities, even if abstract, are preferable for maintaining utility."

Two more findings matter here. The more capable models, Qwen2.5-72B and GPT-4o mini, suffered the largest drops, which the authors read as a stronger reliance on specific entity information. Retrieval tasks (the RGB benchmark) collapsed under anonymization, while TruthfulQA, a benchmark about avoiding confident falsehoods, improved. Anonymization is not uniformly bad. It is bad for tasks that depend on who did what to whom, which is a fair description of most legal work.

Now the limits, here rather than in a footnote. The tasks were medical question answering, multi-step reasoning puzzles and retrieval. No legal document was tested. The paper did not test realistic surrogate names (replacing Ahmad with Bilal), and it did not test role labels such as "Party A". Applying its results to a contract is our inference. We are reasoning from a paper about entity-dependent tasks to a document type that is almost entirely entity-dependent, and we think the step is sound. It is still a step we took, not one the authors did.

Take a 40-page facility agreement with a borrower, two lenders, an agent bank and a guarantor. Replace every company name with [COMPANY] and you have merged five parties into one. The model can no longer tell whether the indemnity runs from the borrower to the agent or from the agent to the lenders, because the clause now reads "[COMPANY] shall indemnify [COMPANY]". Write [BORROWER-1], [LENDER-1], [LENDER-2], [AGENT-BANK-1] and [GUARANTOR-1] instead and you have protected exactly the same names while keeping the structure the answer depends on.

Stop writing [CLIENT]. Write which client.

Why automated redaction cannot be trusted blindly

Presidio is the open source anonymization engine that started at Microsoft, and the first warning on its documentation homepage reads:

Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed.

A vendor being honest about its own limits is the most useful sentence in this section. Automated detection finds patterns: strings shaped like names, numbers shaped like passport numbers. It does not know that a project name identifies your client to anyone in the local property market, or that a costs figure in dirhams can be matched to one published judgment in a minute of searching.

The failures that survive a clean-looking redaction tend to be the same four:

• Metadata. The author field, the creation date, tracked changes and the revision history of a Word file that was exported to PDF. None of it is on the page and all of it is in the file.
• The text layer. A scanned document with a black box drawn over the image still carries the recognised text underneath. That is what the Manafort filing and the Meta slides had in common.
• Numbers and dates that re-identify by context. A contract value and a signing date will often narrow a deal to one candidate.
• Amounts only one matter could have. Costs orders and settlement figures are searchable.

The commercial tools for legal redaction include Relativity, Everlaw, Litera Blackout, Objective Redact, Adobe Acrobat and CaseGuard. Presidio is an open source option you can run on your own machine. We are not ranking them. The point of this post is that the tool matters less than the check afterwards. Any tool will produce a file that looks redacted. Whether it is redacted is a question you answer by opening the output and attacking it yourself: select all, copy, paste into a plain text editor, then read the document properties.

The two duties

Redaction matters because of two obligations that lawyers tend to collapse into one.

Confidentiality is owed to the client. It is the duty that makes you hesitate before pasting a term sheet into a chat window, and it is answered, in part, by removing what identifies the client before you do.

Data residency is owed to the law. It is the question of where the data may physically sit and which regulator has jurisdiction over it. In the DIFC, Regulation 10, which covers personal data processed through autonomous and semi-autonomous systems, was enacted on 1 September 2023 and has been in full enforcement since 1 January 2026. Anonymizing a document reduces what you send. It does not move the AI provider's servers, and it does not answer whether processing personal data through that system is permitted for the matter in front of you.

Anyone selling you redaction as a compliance answer is selling you half of one.

There is a third thing redaction does not do. It does not make the conversation privileged. A federal court in New York ruled in February 2026 that documents a defendant produced with a consumer AI tool were neither privileged nor work product. We covered the ruling, and the door it leaves open for use directed by counsel, in our post on why AI conversations are not privileged. Redaction protects the client's identity. It does not protect the exchange.

A process that works

Five steps, in this order. None of them needs a budget.

• Redact before you prompt, not after. Anything that reaches the model has already left your control, and editing the prompt later does not un-send it.
• Use role-preserving pseudonyms and keep them consistent across the whole document. [LENDER-1] on page 3 is [LENDER-1] on page 38. If a party appears under two names, a full legal name and a defined term, both map to the same label.
• Strip metadata as a separate, explicit step. Export a fresh PDF, then open the properties and look. Do not assume "save as" did it.
• Read the output. Select all, copy, paste into a plain text file. If a redacted term reappears, the redaction was a drawing.
• Keep the mapping somewhere the document is not. The key that turns [LENDER-1] back into a name is what makes the document pseudonymized rather than anonymized. It belongs in the matter file, not in the folder you sent.

We did the same thing to our Lebanese waqf benchmark before publishing it. The family was not ours to name, and the legal structure had to survive intact for the test to mean anything. Both were possible at once. They usually are.

Key takeaways

• A black bar is a drawing. Confirm deletion by trying to defeat your own redaction.
• Redaction, anonymization, pseudonymization and tokenization are four different operations, and only the irreversible ones take the data outside the scope of EU data protection law.
• In the September 2026 study, generalization ([CITY]) and redaction produced the largest performance drops. Entity-unique pseudonyms did best. No legal task was tested, so the contract application is our inference.
• Automated detection has limits its own vendors put in writing. Put a human check on top.
• Anonymization answers confidentiality. It does not answer data residency, and it does not create privilege.

• Deußer et al., On the Impact of Anonymization on the Performance of Large Language Models, arXiv 2609.11335 v2, 15 September 2026
• Columbia Journalism Review, Thank you to everyone who can't redact documents properly, 10 January 2019
• Techdirt, Leaker of those FTC/Microsoft documents revealed: it was Microsoft, 21 September 2023
• The Verge, Google, Apple, and Snap aren't happy about Meta's poorly-redacted slides, 16 April 2025
• Presidio documentation, warning callout on the homepage
• Regulation (EU) 2016/679, the General Data Protection Regulation, Article 4(5) and Recital 26
• DIFC Commissioner of Data Protection, Regulation 10
• Mayer Brown, AI regulation in the DIFC: personal data processed through autonomous and semi-autonomous systems, January 2026