---
title: "Legal-AI-Halluzination: Das gefälschte Zitat, das jede Prüfung besteht"
url: https://www.haqq.ai/de/blog/legal-ai-hallucination-fake-citation
locale: de
type: blog
description: "Wir haben ein Frontier-Modell eine juristische Datenbank imitieren lassen. Es weigerte sich, erfundene Fälle zu erfinden, zitierte dann aber ein echtes Gesetz für den falschen Sachverhalt – die Halluzination, die eine anwaltliche Prüfung übersteht."
category: "ai-legal-tech"
keywords: "legal AI hallucination, AI legal citation hallucination, fake legal citations, do AI legal research tools hallucinate, name cite mismatch, retrieval grounded legal AI, AI fake case citations"
published: 2026-06-30
english_version: https://www.haqq.ai/blog/legal-ai-hallucination-fake-citation
source: src/data/blogPosts.ts
generator: scripts/generate-page-markdown.ts
---

# Legal-AI-Halluzination: Das gefälschte Zitat, das jede Prüfung besteht

Wir haben ein Frontier-Modell eine juristische Datenbank imitieren lassen. Es weigerte sich, erfundene Fälle zu erfinden, zitierte dann aber ein echtes Gesetz für den falschen Sachverhalt – die Halluzination, die eine anwaltliche Prüfung übersteht.

> **Note:** TL;DR: We turned a frontier model (Opus 4.8) into a fake legal database and probed where it fabricates. It refused to invent citations that do not exist, 4 out of 4 impossible references came back not found. But on a plausible query it fabricated the search itself: a ranked result list with a made-up relevance score and a real, clickable citation pointing to the wrong law. The dangerous hallucination is not the fake case you can catch. It is the real citation attached to the wrong content, and the model insisting it looked something up when it did not.

## The number everyone quotes, and the one they miss

If you have shopped for legal AI in the last year, you know the scary statistic. A 2024 [Stanford study](https://arxiv.org/pdf/2405.20362) measured hallucination rates of 43% for GPT-4, 33% for Westlaw's AI research, and 17% for Lexis+. A separate database now tracks [over 1,000 court cases](https://www.llrx.com/2026/02/what-the-science-says-about-hallucinations-in-legal-research/) where lawyers filed AI-invented citations. Some have been fined.

**Hallucination rates in legal research tools**

Stanford, 2024, share of responses containing a hallucination

- GPT-4 (general LLM): 43%
- Westlaw AI research: 33%
- Lexis+ AI: 17%

_Retrieval-grounded legal tools beat general models, but the errors that remain get subtler, like a real citation attached to the wrong law._

The advice that follows is always the same: verify every citation. True, and nearly useless, because it assumes hallucinations look like obvious fakes. The ones that get lawyers sanctioned do not. So we ran an experiment to see what they actually look like.

## The experiment: make the model be the library

Most hallucination tests ask a model a legal question and grade the answer. We did something stranger. We used an open-source tool, [world-model-harness](https://github.com/experientiallabs/world-model-harness), to make Opus 4.8 impersonate the legal database itself, the search engine, not the lawyer.

Here is the setup in plain terms. We recorded real legal-research sessions against a genuine multi-jurisdiction database, statutes and case law across the EU, France, and Germany. Then we trained the model to play that database: you send a query, it returns what it thinks the database would return, search hits, document text, citations. A real database retrieves. Ours predicts. The gap between the two is the hallucination, isolated under a microscope.

> **Note:** A confession from the build: our first run showed almost 100% citation accuracy and we nearly celebrated. Then we realized we were testing the cache, not the model, we had fed it queries it had already seen, so it simply echoed the real answers back. Embarrassing for an afternoon, instructive forever: the only honest test of a legal AI is on questions it has not seen before.

## Finding 1: it will not invent a citation from nothing

We fed the fake library four references that do not exist anywhere: a made-up EU regulation, a fabricated French Code civil article, a nonsense CELEX document number, and a fake German constitutional case.

All four times, it answered the way a real database would: resolved false, not found. It did not invent a plausible case to fill the gap. This is genuinely good news and worth saying out loud. Frontier models in 2026 are far better calibrated than the they-just-make-things-up reputation suggests. Asked for something impossible, Opus 4.8 declined. If that were the whole story, you could relax. It is not.

## Finding 2: it fabricates the search itself

Then we asked it something plausible but unseen, about the EU AI Act's rules for high-risk systems. The model produced a complete, ranked database response: multiple hits, official-journal dates, a relevance score of 0.71243286, all framed as if retrieved from EUR-Lex. None of that retrieval happened. There was no index, no search, no score. It manufactured the act of looking something up. And the citations it returned are the masterclass in why this matters:

- CELEX 32024R1689, real and correct. That genuinely is the EU AI Act. Impressive recall.
- CELEX 32024R0900, real, clickable, and wrong. That citation resolves perfectly on EUR-Lex, to Regulation (EU) 2024/900 on the transparency of political advertising. It has nothing to do with AI conformity.

That second one is the whole point. It is not a fake citation, a fake citation is easy to catch because it does not resolve. This is a real citation attached to the wrong law. Citation-checkers call it a name/cite mismatch. You click it, an official EU regulation loads, everything looks legitimate, and you have just grounded your argument in advertising law while thinking it is AI law. Every link resolves. Nothing looks wrong.

> **Note:** Honest caveat: this is a sharp proof-of-concept, not a benchmark, a small corpus and one fabrication caught across a handful of probes. We are not claiming a rate. We are showing you a shape.

## Why this is the failure that survives review

Go back to the standard advice: verify every citation. Now watch it fail. The lawyer checks the AI Act cite, real. Checks the next one, also real, loads fine. Verification passed, because verification usually means does this link work, and every link worked. The error is not whether the citation exists. It is whether it says what the model claimed, a much harder thing to catch, and exactly the kind of mistake that ends up in a filing.

The deeper lesson is about where the lie lives. We instinctively worry the model will get the law wrong. The real risk is that it gets the retrieval wrong, that the sentence I searched the database and here is what I found is itself the hallucination, complete with a fake confidence score to sell it.

## HAQQ's take: the lookup is the hallucination

This is the reason we ground HAQQ's answers in real retrieval instead of asking a model to recall the law from memory. Not because models are dumb, this one refused every impossible citation and nailed the AI Act's number. But a model asked to be the source will eventually perform the retrieval it did not do. The only fix is to make the retrieval real: pull the actual document from an actual index, then verify the citation resolves to the case the model named, not just to a case.

A buyer's question follows from this. Do not ask a legal-AI vendor whether they hallucinate, everyone says no. Ask: when you show me a citation, did you retrieve that exact document, and do you check that the case name matches the cite? The name/cite mismatch is the tell. A tool that only checks whether citations are well-formed will wave 32024R0900 straight through.

## Key takeaways

- A well-calibrated frontier model will not invent citations from thin air, it refused 4 of 4 impossible references.
- It will fabricate the retrieval itself on plausible queries, including a fake relevance score and a real citation pointing to the wrong law.
- The dangerous hallucination is the one that passes verification, every link resolves, only the meaning is wrong.
- Ground legal AI in real retrieval plus citation-matching, and judge vendors on whether they check that a cite says what they claim, not just that it exists.

## Sources and further reading

- [Stanford, Assessing the Reliability of Leading AI Legal Research Tools (2024)](https://arxiv.org/pdf/2405.20362)
- [What the Science Says About Hallucinations in Legal Research (LLRX)](https://www.llrx.com/2026/02/what-the-science-says-about-hallucinations-in-legal-research/)
- [world-model-harness, the open-source tool we used](https://github.com/experientiallabs/world-model-harness)

## Häufig gestellte Fragen

### Halluzinieren KI-Tools für die juristische Recherche 2026 noch?

Ja, aber die Rate hängt stark von der Architektur ab. Allgemeine Modelle halluzinieren am meisten; Tools, die Antworten in echten Dokumenten verankern (Retrieval-Augmented Generation), halluzinieren weit weniger. Eine Stanford-Studie von 2024 maß 43 % für GPT-4 gegenüber 17 bis 33 % für retrievalbasierte juristische Tools. Die verbleibenden Fehler sind subtiler, etwa ein echtes Zitat, das dem falschen Gesetz zugeordnet ist.

### Was ist ein Name/Zitat-Mismatch?

Es liegt vor, wenn ein Zitat echt ist und zu einem tatsächlichen Dokument führt, aber zu einem anderen Fall oder Gesetz als dem genannten. Es besteht eine naive Prüfung, ob der Link funktioniert, und ist deshalb gefährlicher als ein offensichtlich erfundenes Zitat, weil an der Oberfläche nichts falsch aussieht.

### Reicht Retrieval-Augmented Generation (RAG), um Halluzinationen zu stoppen?

Sie verringert sie drastisch, ist aber keine Zauberei. RAG kann trotzdem ein echtes Dokument liefern und falsch zuordnen. Der entscheidende Schritt ist die Überprüfung: bestätigen, dass das abgerufene Zitat zur Aussage passt, nicht nur, dass es existiert.

### Wie sollte ein Anwalt KI-generierte Zitate überprüfen?

Prüfen Sie, dass jedes Zitat auflösbar ist und dass das Dokument, zu dem es führt, tatsächlich besagt, was die KI behauptet: die Entscheidung, den Gegenstand der Verordnung, die Artikelnummer. Der Mismatch versteckt sich im zweiten Schritt, nicht im ersten.

### Kann man KI juristische Recherche anvertrauen?

Nur mit Überprüfung und der richtigen Architektur. Behandeln Sie KI-Ergebnisse wie die eines schnellen, aber unerfahrenen Rechercheurs: nützlich für Tempo, nie die letzte Instanz. Bevorzugen Sie Tools, die aus echten Quellen abrufen, und prüfen Sie, dass Zitate zur Aussage passen, bevor Sie sich darauf verlassen.

---

Canonical page: https://www.haqq.ai/de/blog/legal-ai-hallucination-fake-citation
Whole-site context for agents: https://www.haqq.ai/llms.txt · https://www.haqq.ai/agents.md

Generated at build time from the site's own content data. Do not edit by hand.
