TL;DR: A real legal matter is not one prompt and one answer. It means reading long documents, reasoning across jurisdictions, finishing multi-step tasks, citing sources that exist, and knowing when to decline. A single-prompt test measures a thin slice of that. This post sets out what a legal AI has to do, and how HAQQ thinks about evaluating it. It contains no scores.
A matter is not a prompt
Vyzkoušejte HAQQ AI zdarma
Zažijte sepisování a rešerše s podporou AI
Ask a lawyer what makes a matter hard and they will rarely say the question. They will say the file: a stack of documents, three different dates, two governing laws, a counterparty who changed position halfway through, and a partner who needs a defensible answer by the end of the day. The difficulty is in the work around the question.
Five things recur. Each one is a place where a legal AI can look fluent and still fail.
Long documents
A data room, a contract bundle or a court file is far longer than anything a person reads in one sitting. The risk is not that the model skips it. The risk is that it reads all of it and weighs the wrong part: a change-of-control clause buried in an annex, a carve-out in a definition, an amendment that quietly overrides an earlier term. A summary that reads well can still miss the one clause the client would be unhappy to learn about later.
Cross-jurisdiction reasoning
Law does not transfer cleanly. A US-style financing instrument that converts into preferred shares assumes a share-class machinery that a Lebanese limited liability company does not have. The doctrine of consideration, central in common law, plays no such role in Saudi contract law. There is no at-will employment in the Gulf states, where notice periods and end-of-service entitlements apply instead. An answer that applies the wrong system is not slightly off. It is built on a rule that does not govern.
So the system has to know which law applies, say so, and keep that law's rules separate from the ones it learned most about.
Multi-step tasks
Diligence, a contract review against a playbook, a research memo with authorities: these are sequences. Find the documents, extract the facts, test them against the law, reconcile the conflicts, draft, check. In our own experiment on a document data room, the same model, the same documents and the same scoring produced different results depending on how the work was structured, not on which model ran it. We wrote that up in single prompt versus a swarm of agents on M&A diligence.
Citation accuracy
A fluent answer attached to a citation that does not exist is the failure courts have been sanctioning. The subtler version is a real passage cited to the wrong article, because it reads as genuine and passes a casual check. Some courts now state the duty plainly. Qatar's QICDRC Practice Direction No. 1 of 2026, for example, requires every citation to be checked against primary sources or a recognised legal database (paragraph 16). We covered it in what the Qatar practice direction actually requires.
The duty to verify does not change with the rules on disclosure. A legal AI therefore has to make the check easy to do and easy to show.
Calibrated refusal
Sometimes the right output is not an answer. The source is missing, the law asked about does not govern, or the question needs facts nobody supplied. A system that always answers will sometimes invent. A system that always declines is useless. Calibration means declining when the evidence is not there and answering when it is, and it is the hardest of the five to test, because a refusal looks like a failure on a simple scoreboard.
Why single-prompt benchmarks miss this
- They hand over the relevant text. Real work starts with finding it.
- They score one answer. Real work is a sequence in which an early error carries forward.
- They reward answering. A model that never declines can score well while being unsafe to rely on.
- They average. One fabricated citation among fifty correct ones is a different event from fifty-one plausible answers, and a mean hides that.
- They are usually written for one legal system, so they cannot show that a model applied the wrong one.
How HAQQ thinks about evaluating it
We treat these as separate dimensions and report them separately, instead of folding them into one number. Where an item can be checked by exact comparison, such as the instrument, the article or a genuine-or-fabricated verdict, scoring is deterministic and uses no model as judge. Where the answer is open-ended, a human reads it. Answer keys are built from primary sources and checked by a lawyer, and some matters carry planted issues so we know what a good answer must catch.
We also include questions designed to be declined, and questions set in one jurisdiction that tempt a system toward the law of another. The scope and method of our MENA work are set out in the MENA legal AI benchmark: scope and method.
Key takeaways
- Hard legal work lives in the file around the question: long documents, several laws, many steps.
- Citation accuracy and calibrated refusal are separate skills, and neither shows up in a single averaged score.
- A useful evaluation reports dimensions separately and states where it cannot yet judge.
