Complex Legal Work: What a Legal AI Actually Has to Do
Long documents, several jurisdictions, multi-step tasks, exact citations and the judgement to decline. What a legal AI has to do on a real matter, why single-prompt tests miss it, and how HAQQ thinks about evaluating it.
FAQ
Why do single-prompt benchmarks miss complex legal work?
They usually hand over the relevant text, score one answer, reward answering over declining, average away rare but serious errors such as a fabricated citation, and are written for one legal system. Real matters involve finding the source, a sequence of steps and the right law.
What is calibrated refusal in legal AI?
It means declining when the evidence is not there, for example when a source is missing or the law asked about does not govern, and answering when it is. A system that always answers will sometimes invent, and one that always declines is not useful.
How does HAQQ think about evaluating complex legal work?
As separate dimensions reported separately, not one number. Items that can be checked by exact comparison are scored deterministically with no model as judge, open-ended answers are read by a person, and answer keys are built from primary sources and checked by a lawyer.
Does the duty to verify citations depend on disclosure rules?
No. Qatar's QICDRC Practice Direction No. 1 of 2026, for example, requires every citation to be checked against primary sources or a recognised legal database, while pre-emptive declaration of AI use is not required unless the court asks.