Benchmark Legal AI pro MENA: rozsah a metoda

Why test by jurisdiction

Most public legal AI benchmarks are built on English-language, common-law tasks. Legal work in the Middle East and North Africa runs on codified law, in Arabic and English, and it differs from one state to the next. Financial free zones such as DIFC and ADGM add their own laws on top of onshore law. A model that answers a Lebanese contract question with a rule from another system has not made a small error. It has applied a law that does not govern the matter.

A single regional score would hide exactly that failure. So the evaluation is organised by jurisdiction, and every item is tied to the law of one place.

Scope: jurisdictions and languages

The evaluation covers eleven jurisdictions: Lebanon, the UAE, Iraq, Egypt, Oman, Jordan, Qatar, Kuwait, Syria, Saudi Arabia and Bahrain. Items are in Arabic and English. In the current prompt set, for example, Egyptian questions are in Arabic and UAE questions are in English, which is how that law is published and practised.

Coverage is not equal across the eleven. Some jurisdictions have more verified source material than others, and the gap list below says which.

What the evaluation asks

• Scenario questions. A realistic matter, written the way a lawyer would pose it, that asks for advice. Practice areas include contract, procedure, criminal, corporate, insolvency, commercial, property, family, banking, insurance, arbitration, employment, evidence, maritime and competition.
• Source attribution. The model sees the verbatim text of one provision and must name the jurisdiction, the instrument and the article it comes from.
• Citation verification. The model sees a passage with a citation attached and must say whether the citation is genuine. Some are real passages attached to the wrong article, which is the failure that matters in practice.
• Out-of-jurisdiction traps. Questions where the right move is to decline, or to say that the law being applied does not govern, rather than to answer in generic terms.

How answer keys are built

Every key starts from a primary source: the statute, decree or gazette text itself, not a summary and not another model's answer. A key records the instrument and article a correct answer must cite, the points a sound answer must cover and, for traps, that declining is the correct response.

A lawyer then checks the key before it is used. That step is what separates a test of the law from a test of agreement with a model.

We also record whether each instrument is in force, using dates rather than a yes or no. A statute can be enacted and published and still not be law: Egypt's Criminal Procedure Code, Law 174 of 2025, was published before its entry-into-force date. Where nobody has checked a status against a gazette or official source, we record it as unknown instead of letting a model guess.

Where an item can be scored by exact comparison, such as the instrument, the article number or a genuine-or-fabricated verdict, scoring is deterministic and uses no model as judge. Run it twice and the score is the same. Open-ended answers are harder: string checks verify grounding and refusal well but only approximate substance, so we do not present an automatic substance number as more than it is.

Where source coverage is thin today

A benchmark is only as good as the primary sources behind its keys. These four jurisdictions are thin today, and we would rather show the gap than a score on weak ground.

• Saudi Arabia. Much of the material we can index is financial-regulator rulebooks, which is narrower than the core statutes.
• Bahrain. We have collections of legislation, but we have not yet checked them to the standard an answer key needs: current, consolidated and traceable to an official page.
• Iraq. The legislation we can index leans toward the Kurdistan Region, so federal coverage is thinner.
• Syria. Our collection is small.

For these jurisdictions, expect fewer items, or none, until the sources are verified. We will update this list when that changes.

Results

No results are published in this post. They will follow the next run, with the scope and method described here, and any change to either will be stated alongside them.

What did the earlier civil-law run (May 2026) show?

This evaluation grew out of a smaller one. In May 2026 we open-sourced HAQQ Legal Benchmark v0: 16 tasks, twelve real matters across the UAE, DIFC, Saudi Arabia, Lebanon, Egypt and Qatar plus four out-of-jurisdiction traps. We built it because the public legal AI benchmarks we could find (Stanford's LegalBench, Harvey's LAB) are written for common-law work, while most of the world's states, and every jurisdiction in this evaluation, run on codified civil law with the statute as the primary source.

The traps were the median mistakes of a common-law model on MENA matters: converting a SAFE into preferred shares of a Lebanese SARL under Delaware's DGCL section 151, checking an English-law doctrine of consideration on a Saudi contract, applying California non-compete rules to a Dubai employee, and advising under the GDPR on a purely domestic UAE matter.

The v0 rubrics were deterministic string checks with no model as judge, which is why the substance score stayed flat for both agents: string checks can verify grounding and refusal, and can only approximate substance. The eleven-jurisdiction evaluation above keeps that rule for exact-match items and adds lawyer-checked keys for the rest.

Key takeaways

• Eleven jurisdictions, Arabic and English, tested one jurisdiction at a time.
• Answer keys come from primary sources and are checked by a lawyer.
• Exact-match items are scored deterministically, with no model as judge.
• Saudi Arabia, Bahrain, Iraq and Syria are thin today, and the gap is stated, not hidden.