TL;DR: The HAQQ MENA legal evaluation tests legal AI on the law of eleven jurisdictions, in Arabic and English, against answer keys built from primary sources and checked by a lawyer. This post covers scope and method only. No results are published here: they follow the next run. Four jurisdictions have thin source coverage today, and we list them below.
Why test by jurisdiction
Încercați HAQQ AI gratuit
Experimentați redactarea juridică și cercetarea asistate de AI
Most public legal AI benchmarks are built on English-language, common-law tasks. Legal work in the Middle East and North Africa runs on codified law, in Arabic and English, and it differs from one state to the next. Financial free zones such as DIFC and ADGM add their own laws on top of onshore law. A model that answers a Lebanese contract question with a rule from another system has not made a small error. It has applied a law that does not govern the matter.
A single regional score would hide exactly that failure. So the evaluation is organised by jurisdiction, and every item is tied to the law of one place.
Scope: jurisdictions and languages
The evaluation covers eleven jurisdictions: Lebanon, the UAE, Iraq, Egypt, Oman, Jordan, Qatar, Kuwait, Syria, Saudi Arabia and Bahrain. Items are in Arabic and English. In the current prompt set, for example, Egyptian questions are in Arabic and UAE questions are in English, which is how that law is published and practised.
Coverage is not equal across the eleven. Some jurisdictions have more verified source material than others, and the gap list below says which.
What the evaluation asks
- Scenario questions. A realistic matter, written the way a lawyer would pose it, that asks for advice. Practice areas include contract, procedure, criminal, corporate, insolvency, commercial, property, family, banking, insurance, arbitration, employment, evidence, maritime and competition.
- Source attribution. The model sees the verbatim text of one provision and must name the jurisdiction, the instrument and the article it comes from.
- Citation verification. The model sees a passage with a citation attached and must say whether the citation is genuine. Some are real passages attached to the wrong article, which is the failure that matters in practice.
- Out-of-jurisdiction traps. Questions where the right move is to decline, or to say that the law being applied does not govern, rather than to answer in generic terms.
How answer keys are built
Every key starts from a primary source: the statute, decree or gazette text itself, not a summary and not another model's answer. A key records the instrument and article a correct answer must cite, the points a sound answer must cover and, for traps, that declining is the correct response.
A lawyer then checks the key before it is used. That step is what separates a test of the law from a test of agreement with a model.
We also record whether each instrument is in force, using dates rather than a yes or no. A statute can be enacted and published and still not be law: Egypt's Criminal Procedure Code, Law 174 of 2025, was published before its entry-into-force date. Where nobody has checked a status against a gazette or official source, we record it as unknown instead of letting a model guess.
Where an item can be scored by exact comparison, such as the instrument, the article number or a genuine-or-fabricated verdict, scoring is deterministic and uses no model as judge. Run it twice and the score is the same. Open-ended answers are harder: string checks verify grounding and refusal well but only approximate substance, so we do not present an automatic substance number as more than it is.
Where source coverage is thin today
A benchmark is only as good as the primary sources behind its keys. These four jurisdictions are thin today, and we would rather show the gap than a score on weak ground.
- Saudi Arabia. Much of the material we can index is financial-regulator rulebooks, which is narrower than the core statutes.
- Bahrain. We have collections of legislation, but we have not yet checked them to the standard an answer key needs: current, consolidated and traceable to an official page.
- Iraq. The legislation we can index leans toward the Kurdistan Region, so federal coverage is thinner.
- Syria. Our collection is small.
For these jurisdictions, expect fewer items, or none, until the sources are verified. We will update this list when that changes.
Results
No results are published in this post. They will follow the next run, with the scope and method described here, and any change to either will be stated alongside them. For the earlier, smaller civil-law benchmark, see the civil-law legal AI benchmark.
Key takeaways
- Eleven jurisdictions, Arabic and English, tested one jurisdiction at a time.
- Answer keys come from primary sources and are checked by a lawyer.
- Exact-match items are scored deterministically, with no model as judge.
- Saudi Arabia, Bahrain, Iraq and Syria are thin today, and the gap is stated, not hidden.
