Confidentiality Is the Next Hallucination: AI Client Confidentiality for Lawyers

The curve we have already watched once

In June 2023, Judge P. Kevin Castel of the Southern District of New York sanctioned the lawyers and the law firm in Mata v. Avianca for a filing that cited court opinions which did not exist. The cases had come from ChatGPT. The penalty was $5,000.

It turned out to be an early entry in a docket. Damien Charlotin's AI Hallucination Cases database had logged 2,077 decisions worldwide as of 23 September 2026, and we keep a running read of it in our hallucination tracker. The shape: one loud case, a stretch where everyone says "not us", then a steady flow once judges know what to look for.

Confidentiality is earlier on the same curve

A fake citation sits in a filing where a judge can check it. A confidentiality failure leaves no such trace unless it resurfaces in discovery, a dispute or a regulator's file. Quieter is not the same as smaller.

So we wrote down the dated, public events, each from a primary source or a major outlet (the DIFC date is from the law firm CMS):

Count the last column: of nine events, one imposed a fine, and a court annulled it. The triggers are visible; the price has barely started. That is roughly where hallucinations stood in 2023.

Courts: privilege first, protective orders next

In February 2026, Judge Jed Rakoff of the Southern District of New York held that roughly 31 documents a criminal defendant had generated with Claude, which the court called a "publicly available AI platform", were protected by neither attorney-client privilege nor the work-product doctrine. (Privilege protects confidential lawyer-client communications made to obtain legal advice. Work product protects material prepared by or for counsel in anticipation of litigation.) The court pointed to the privacy policy that users of Claude consent to (it cited the 19 February 2025 version), under which Anthropic collected inputs and outputs, used such data to train Claude and reserved the right to disclose it to third parties. He left one door open: had counsel directed the use, Claude "might arguably" have acted like a professional working as the lawyer's agent, a line that leads back to the 1961 Kovel decision. Our full write-up of the ruling covers the rest.

Privilege decides what can be kept out of evidence. The duty of confidentiality is wider: in the Florida Bar's words, it "applies to all information learned during a client's representation, regardless of its source." That wider duty is where the next cases sit.

A protective order is the court order that controls how each side may use documents produced in discovery. On 30 March 2026, in Morgan v. V2X, a magistrate judge in Colorado amended one so that no party may put material designated confidential into an AI platform "unless the AI provider is contractually prohibited from: (1) storing or using inputs to train or improve its model; and (2) disclosing inputs to any third party except where such disclosure is essential to facilitating delivery of the service." The provider must allow deletion on request, and a party relying on a tool "must retain written documentation of these contractual protections." The judge said plainly that, for now, the clause will keep "most, if not all," mainstream free and low-cost AI tools away from confidential material.

Breaching a protective order already has a price. In November 2025, in Valjakka v. Netflix, a federal court in California ordered a lawyer and his firm to pay $95,000 in compensatory sanctions for sharing Netflix's confidential documents with a third-party firm the order did not cover. No AI was involved: the recipient was a litigation-funding firm, not a chatbot. Put that next to the Morgan clause and the distance from "uploaded it to a chatbot" to "sanctioned" is short.

Regulators: Rome and the DIFC

On 31 March 2023, the Garante ordered a temporary limitation on OpenAI's processing of Italian users' data, noting a 20 March data breach involving users' conversations and subscribers' payment information. In December 2024 it fined OpenAI EUR 15 million, finding among other things that the breach had not been notified and that users' data had been used to train ChatGPT without an adequate legal basis. A Rome court suspended the fine in March 2025 and annulled it in March 2026. The fine did not survive. The question behind it, whether user data may train a model, is one clients now answer in their own guidelines.

For Gulf firms, the text in force is Regulation 10 of the DIFC Data Protection Regulations, enacted on 1 September 2023, which covers personal data processed through autonomous and semi-autonomous systems, meaning AI. According to the law firm CMS, full enforcement was set to begin in January 2026. Deployers and operators must give notice at initial use of any application or website service that uses such systems to process personal data. For a DIFC firm, client personal data inside an AI tool is now a regulated processing question as well as an ethics one.

Clients: Samsung, then outside-counsel guidelines

In May 2023, Bloomberg reported that Samsung was restricting staff use of generative AI after engineers accidentally leaked internal source code by uploading it to ChatGPT. The memo worried that such data is stored on external servers, "making it difficult to retrieve and delete." Samsung is not a law firm, but it is the kind of client firms work for.

Outside-counsel guidelines are the rules a client gives the firms it hires. Some now cover AI. Zscaler's, revised on 7 April 2025: "Zscaler data or confidential information cannot be used to train AI tools, whether proprietary or third-party." The FDIC's Outside Counsel Deskbook, revised in April 2026, requires written notice before using generative AI on an FDIC matter and says firms must "Never enter Sensitive Information or FDIC work product into a GenAI tool that operates in an 'open environment.'" Its footnote, in the FDIC's own words, gives "many programs freely accessible for public and commercial use such as ChatGPT, Gemini, Claude, LLaMA, Grok, and Stable Diffusion" as examples of GenAI operating in an open environment.

How common is it? In a study dated 16 September 2026, Fulkerson Advisors searched 1,054 published outside-counsel guidelines written since ChatGPT's release. Twenty mention AI; twelve of those require disclosure of AI use, and nine require keeping client information out of public AI tools. It is one consultancy's sample, limited to what search engines and procurement portals surface, and it leans on public bodies and universities, since most organizations do not publish theirs. The trend still points one way: none dated before 2024 mention AI, then 2 of 220 dated 2024, 8 of 365 dated 2025 and 10 of 261 dated 2026. By our arithmetic: 0.9%, 2.2%, 3.8%.

What six pieces of ethics guidance already require

Our own count. We read ABA Formal Opinion 512 and the five state and DC bar opinions and guidance documents on generative AI that it cites, and recorded what each says about consent before client information goes into an AI tool:

The tally: all six tie client information in an AI tool to the client's consent, to adequate protections, or to both. Two make consent mandatory in so many words (the ABA and California). Pennsylvania bars confidential input into tools without adequate protections. West Virginia wants consent confirmed in writing. Florida calls it a recommendation. DC adds that clients "typically should not be asked to consent" to wide-ranging disclosures. And the ABA closes the shortcut: "merely adding general, boiler-plate provisions to engagement letters purporting to authorize the lawyer to use GAI is not sufficient." (We quote California's current 2026 text, which replaced the 2023 version the ABA cited.)

Our prediction

This is a forecast, not a finding.

We expect damages or sanctions for an AI-related confidentiality breach. Heppner shows AI chat records can be seized and read, Morgan shows a court writing an AI test into a protective order, Valjakka shows what breaching one costs, and six pieces of ethics guidance define the duty. The missing piece is the public case that joins them. We expect it to go the way Mata did: one loud case, then a docket.

We expect clients to write AI terms into engagement letters and outside-counsel guidelines. No-training clauses and notice duties are already there. The next step, we think, is proof: which account a document went into and what was redacted first. None of the guidelines we quoted asks for that yet.

What to put in place now

1. A specific AI clause in the engagement letter

The ABA calls the engagement agreement "a logical place to make such disclosures and to identify any client instructions on the use of GAI in the representation," and boilerplate does not count as consent. Write it like a scope clause: name the tools and account tier, what goes in only after redaction, what never goes in, who reviews the output, and how the client can opt out. If a matter needs identifiable data inside a tool, get a separate consent for it.

2. Redact before upload, and keep a log

The NYC Bar's Formal Opinion 2024-5 says "Consent is not needed if no confidential client information is shared, for example through anonymization of client information." That makes redaction the cheapest control here, with a caveat: swapping names is pseudonymisation, not anonymisation, and context can still identify a client. Our guide to redaction tools for lawyers covers the method, and deciding what goes in before you prompt makes it routine.

Then keep a log, one line per document: date, matter, tool, account, categories replaced, who checked, and where the key to the placeholders lives. Morgan asks for paper on a tool's terms; a redaction log is paper on what you sent through it.

3. Four questions for every tool, answered in writing

ABA Formal Opinion 512 says lawyers should read and understand the terms of use and privacy policy of any generative AI tool they use, and California says reasonable efforts "require more than reliance on generalized marketing assurances." So ask:

• Training: is the provider contractually barred from training on our inputs?
• Retention: how long are inputs and outputs kept, and can we delete them on request?
• Subprocessors: who else receives the data, on what terms?
• Location: where is it processed and stored? For a DIFC firm, that is also a Regulation 10 question.

The Morgan clause is a ready-made test: no training, no disclosure beyond service delivery, deletion on request, documentation on file. Ask every vendor, HAQQ included, for those answers in writing.

4. Know which account tier you are on

Anthropic's August 2025 consumer-terms update applies to Claude Free, Pro and Max and not to services under its Commercial Terms, such as Claude for Work and the API. Consumer users choose whether their data is used to improve Claude; allowing it means five-year retention, declining keeps the 30-day period. Part of the reasoning in Heppner was the privacy policy that users of Claude consent to, and the Morgan court noted that enterprise-tier accounts meeting its test may be available only through organisational procurement or at costs a pro se litigant is unlikely to bear. If you do serious legal work on Claude, check which terms your account sits under before a client asks.

Where HAQQ fits today

Our contribution here is small and free. The HAQQ redaction tool runs in your browser, swaps names, companies, amounts and IDs for typed placeholders you confirm, and lets you download the redacted text and the mapping file separately, for use with any AI tool. Its page says what it is not: "The output is pseudonymised, not anonymised." We would rather you read that from us than from opposing counsel. For the firm-wide version, chapter 30 of the HAQQ Academy course covers rolling AI into a firm without breaking privilege.

Key takeaways

• Hallucination cases went from a curiosity in 2023 (Mata v. Avianca) to 2,077 logged decisions by 23 September 2026. Confidentiality is earlier on the same curve.
• Of nine dated confidentiality events since 2023, one imposed a fine, and a court annulled it. The triggers are visible; the price is still to come.
• Courts have moved from privilege (Heppner) to protective orders with contractual tests for AI tools (Morgan v. V2X).
• Clients already write AI rules into outside-counsel guidelines; all six pieces of ethics guidance we read tie client data in AI tools to consent, adequate protections, or both.
• Put four controls in place now: a specific AI clause, redaction with a log, written answers on training, retention, subprocessors and location, and the right account tier.

Sources and further reading

• Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 22 June 2023)
• Damien Charlotin, AI Hallucination Cases database
• United States v. Heppner, No. 25 Cr. 503 (JSR) (S.D.N.Y. 17 February 2026), opinion text
• Morgan v. V2X, Inc., No. 1:25-cv-01991 (D. Colo. 30 March 2026), order amending the protective order
• Valjakka v. Netflix, No. 4:22-cv-01490 (N.D. Cal. 19 November 2025), order on sanctions
• ABA Formal Opinion 512, Generative Artificial Intelligence Tools (29 July 2024)
• State Bar of California, Practical Guidance for the Use of Generative AI in the Practice of Law (2026)
• Florida Bar Ethics Opinion 24-1 (19 January 2024)
• Pennsylvania and Philadelphia Joint Formal Opinion 2024-200 (22 May 2024)
• West Virginia Legal Ethics Opinion 24-01 (14 June 2024)
• DC Bar Ethics Opinion 388
• NYC Bar Formal Opinion 2024-5
• Garante per la protezione dei dati personali, ChatGPT limitation (31 March 2023)
• Garante per la protezione dei dati personali, ChatGPT decision and fine (20 December 2024)
• ANSA, Rome court annuls the Garante's fine on OpenAI (20 March 2026)
• DIFC Commissioner of Data Protection, Regulation 10
• CMS, Global cyber expectations for 2026, part 2 (12 March 2026)
• Mayer Brown, AI regulation in the DIFC (19 January 2026)
• Bloomberg, Samsung Bans Staff's AI Use After Spotting ChatGPT Data Leak (2 May 2023)
• Zscaler Outside Counsel Billing Guidelines (revised 7 April 2025)
• FDIC Outside Counsel Deskbook (revised April 2026)
• Fulkerson Advisors, What Clients Now Tell Their Law Firms About AI (2026), CC BY 4.0
• Anthropic, Updates to Consumer Terms and Privacy Policy (28 August 2025)
• HAQQ: AI hallucination cases tracker
• HAQQ: Are AI chats privileged? A federal court says no
• HAQQ: How to anonymize a document before you give it to AI