Why General AI Gets Kuwaiti and Lebanese Law Wrong (and How to Test Any Tool)
Lawyers in Kuwait and Lebanon tell us the same thing: general chatbots sound right and cite the wrong law. Why that happens, and a five-question test you can run on any legal AI tool.
Two conversations with lawyers this month: one handling Kuwaiti and Egyptian work, one at a small firm in Lebanon. Both used general chatbots. Both said the same thing in different words: it's fine for rewording, and it gets the law wrong.
The Kuwait-side lawyer put it bluntly: on Kuwaiti law, the answers were simply wrong. The lawyer in Lebanon still uses a paid chatbot every day, but only in pieces: fix this paragraph, explain this article. Never the whole pleading.
They're not being fussy. There are structural reasons general AI struggles with Arab jurisdictions.
The short answer
- General models guess fluently. A Stanford study found general chatbots hallucinated on 58% to 88% of specific, verifiable questions about US federal cases, and couldn't reliably tell when they were doing it.
- US federal case law is among the most documented law on the internet. Kuwaiti cassation principles and Lebanese chamber decisions are far less available in open, machine-readable form. Expect more guessing, not less. (Our inference, not a measured rate.)
- Arabic legal language is its own domain. Research benchmarks for Arabic legal knowledge only started appearing in 2024.
- Test before you trust. Five local questions will tell you more than any marketing page, ours included.
Four reasons general AI misses local Arab law
1. Thin training data
A model learns law the way it learns everything: from text it has seen. Where judgments are published openly and in volume, it has more to learn from. Where decisions sit in subscription collections, court registries or paper bulletins, it has less. It still answers. It just fills the gaps with plausible text.
2. Borrowed law
Many Arab codes share roots: the Egyptian civil code tradition, French influence in Lebanon, cross-borrowing across the Gulf. That's useful for comparative work and dangerous for a chatbot, which may give you a correct Egyptian rule and label it Kuwaiti. Same concept, different article number, sometimes a different outcome.
3. Amendments and timing
Laws change. A model trained on older text will confidently quote an article as it stood before an amendment. The lawyer in Lebanon raised a live example: the status of a general amnesty that kept shifting week to week. A model frozen months ago can't know.
4. It doesn't know when it doesn't know
The Stanford team found models "cannot always predict, or do not always know, when they are producing legal hallucinations." That's the real problem. A wrong answer that says "I'm not sure" is safe. A wrong answer delivered with an article number is not.
A five-question test for any legal AI tool
Run these on any tool you're considering. Use questions where you already know the answer.
Score it on two things: was it right, and could you check it? A tool that's right but shows no sources still leaves you doing the research.
For what a complete citation looks like in each country, see How to Check an AI Legal Citation Before You File It.
Where HAQQ Legal AI fits
HAQQ Legal AI is built for legal work in Arab and international jurisdictions, applies the jurisdiction you select, and cross-checks verified sources with a reference beside each answer. We'd still rather you run the five questions on it than take our word for it.
Frequently asked questions
Is a paid chatbot plan more accurate on local law?
Paid plans give you newer models and features like projects and file uploads. They don't change where the model's legal knowledge comes from. Run the test.
Can I fix this by uploading the law myself?
Partly. Giving a model the actual text you rely on reduces guessing on that text. It doesn't help with the case law you didn't upload.
Should I stop using general AI for legal work?
Use it for what it's good at: structure, wording, summaries of documents you provide. Don't use it as a source of law without checking.
Sources and further reading
- Dahl, Magesh, Suzgun and Ho, "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models" (Journal of Legal Analysis, 2024; arXiv 2401.01301)
- Hijazi et al., "ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models", ArabicNLP 2024
FAQ
Is a paid chatbot plan more accurate on local law?
Paid plans give you newer models and features like projects and file uploads. They don't change where the model's legal knowledge comes from. Run the test.
Can I fix this by uploading the law myself?
Partly. Giving a model the actual text you rely on reduces guessing on that text. It doesn't help with the case law you didn't upload.
Should I stop using general AI for legal work?
Use it for what it's good at: structure, wording, summaries of documents you provide. Don't use it as a source of law without checking.