In short: we gave nine AI systems one real Arabic instruction from a Lebanese client setting up a family endowment, then scored every answer against the 46 articles of the law of 10 March 1947 rather than against each other. Justinian V2 scored 88, our current engine 83, and the highest-scoring general-purpose model 82. The number that matters is not the total. It is Article 10, which decides where the property goes when the endowment ends, and which most of the field either skipped or answered in a way that would have promised the client something the statute does not allow.
Everyone benchmarks legal AI on exam questions
There is real academic work on Arabic legal AI now. ArabLegalEval built a multitask benchmark from Saudi Ministry of Justice documents. IslamicLegalBench evaluates models across seven schools of jurisprudence and 1,200 years of texts. Another team benchmarked Islamic inheritance reasoning specifically.
All of it is multiple choice, or close to it. A model picks A, B, C or D, and a score comes out. That tells you something real about knowledge. It tells you almost nothing about what happens when a client sends a lawyer four hundred words of Arabic with typos in it and asks for an opinion.
So we ran the second thing. One instruction, as it arrived. Nine systems. And instead of grading against a marking scheme, we pulled the governing statute and checked every citation, article by article, against the primary text.
Why this is the harder test. In a multiple-choice benchmark, a wrong answer costs a mark. In a client matter, a wrong answer becomes a deed. The failure mode we care about is not a model that says "I don't know". It is a model that writes a confident, well-formatted opinion promising an outcome the law forbids.
The matter
A Lebanese client, Sunni, wants to set up what Arabic law calls a waqf dhurri, a family endowment. For readers who have not met the structure: property is frozen out of ordinary ownership. It cannot be sold, gifted or inherited. Its income goes to people the founder names. The property itself, the corpus, goes somewhere else again when the arrangement ends.
The instruction asked for five things. Income to a son and a daughter for their lifetimes. When one dies, the whole income to the survivor. The founder administers it while he lives, then two of his other children administer it jointly. No administrator may ever take a fee. And when both income beneficiaries have died, the endowment is wound up and the property split equally between his two other sons, or their children.
That last sentence is the whole problem, and eight of the nine systems did not see it.
We have anonymised the parties throughout. The instruction was a real one and the family is not ours to publish.
How we scored it
We pulled the full text of the Lebanese family endowment law of 10 March 1947, published in Official Gazette number 11 on 12 March 1947, from the Lebanese University's Legal Informatics Centre. Forty-six articles. Then we read every citation each system produced and checked it against the article it claimed.
This matters more than it sounds. A benchmark where the grader has not read the statute is a vibe check with a number attached. Several systems in this run cited articles that exist and said things about them that the article does not say. You cannot catch that by comparing outputs to each other, because the outputs agree with each other more than they agree with the law.
Six dimensions, 100 points. Verified grounding and issue-spotting carry 20 each, as do actionability and safety of the advice. Fact-gap discipline and signal density carry 10 each. Only one of those, verified grounding, is scored directly against primary text. It is the one we would defend hardest.
The results
| System | Grounding /20 | Issue-spotting /20 | Actionability /20 | Safety /20 | Fact-gap /10 | Density /10 | Total |
|---|---|---|---|---|---|---|---|
| Justinian V2 (coming soon) | 19 | 19 | 16 | 18 | 8 | 8 | 88 |
| Justinian (current) | 17 | 17 | 18 | 16 | 7 | 8 | 83 |
| Grok 4.7 | 16 | 16 | 18 | 15 | 8 | 9 | 82 |
| DeepSeek-V4 Pro | 15 | 13 | 15 | 14 | 7 | 8 | 72 |
| GPT-6 Astra | 11 | 8 | 8 | 13 | 9 | 5 | 54 |
| Claude Opus 5 | 7 | 12 | 12 | 9 | 6 | 7 | 53 |
| Claude Fable 5 | 5 | 8 | 9 | 6 | 4 | 6 | 38 |
| Gemini 3.1 Pro | 2 | 2 | 10 | 3 | 1 | 3 | 21 |
A five-point spread across the top three is inside the noise of a single run, and we are not going to pretend otherwise. Grok 4.7 was excellent. It quoted Article 8 word for word and cited the Official Gazette number correctly, which no other general model did.
The interesting separation is lower down, and it is on one dimension.
Verified grounding: citations checked against the statute
This dimension alone is scored directly against primary text. Each citation was read against the article it claimed.
Two systems scored under 6. One cited no article of the governing law at all and told the client the arrangement raised no issue.
Article 10, and why almost everyone missed it
The client wants the property split equally between two of his sons when the endowment ends. Neither of those sons receives income from it. They are not beneficiaries, they are the people he wants to end up owning the thing.
Article 10 of the 1947 law says where the property actually goes:
ينتهي الوقف الذري بانتهاء الطبقة او الطبقتين, ويرجع الوقف الى ملكية الواقف ان كان حيا والى ورثته من الطبقة الاولى او الثانية حسب الاحوال ان كان ميتا.
The endowment ends when the beneficiary tier or tiers end, and the property returns to the founder if he is alive, or to his heirs from the first or second tier if he is dead.
The destination is fixed by the statute, not by the founder's wish. Naming two people in the deed does not put the property in their hands if the law sends it somewhere else. On top of that, Article 27 carves fifteen percent off any pure family endowment at partition and hands it to the local endowment authority for charitable purposes, which means the client cannot receive an equal split of one hundred percent of anything.
Of the nine systems, four wrote the equal split into their advice as though including it in the deed were enough. One reported the arrangement as conforming, in its words, entirely to Lebanese Sunni practice, without citing a single article of the governing law.
Justinian V2 alone stated Article 10 and refused the promise. Its wording, translated: naming the two sons alone to own all the assets equally requires proof that this is consistent with that provision, and simply writing the two names into the deed is not sufficient.
Three more articles that decide this matter
Once you are reading the statute rather than around it, the matter turns out to have several more traps.
- Article 30. The sharia court is required to remove an administrator who is not himself a beneficiary and appoint one who is. The client's chosen administrators receive no income, so the arrangement he asked for is exposed to judicial rejection on its face.
- Article 40. Where an endowment is arranged in tiers, a deceased beneficiary's share passes to his own branch. That may defeat the survivorship the client wants, which sends the whole income to the other child instead.
- Article 9. The founder is excluded from the count of tiers, which changes how the two-tier ceiling in Article 8 applies to this structure.
- Article 42. An administrator is liable for slight negligence only if he is paid. The client's insistence that nobody take a fee quietly lowers the standard of care he can enforce.
Article 30 was caught by our current engine and by V2. Articles 9, 40 and 42 were caught by V2 alone. No general-purpose model reached any of them.
Essayer HAQQ AI gratuitement
Découvrez la rédaction et la recherche juridique par IA
What actually separates a legal AI from a good general model
Grok 4.7 finished one point behind our shipping product. That should tell you something honest about the state of the market: a frontier general model with web search, pointed at a well-indexed statute, gets most of the way there.
What it did not do was ask what the statute does to the client's instruction. It answered the question as asked. The difference in this run was not knowledge of Lebanese law. It was the disposition to treat the client's wish as a hypothesis to be tested against mandatory rules, rather than a specification to be drafted.
That disposition is the product. Everything else, the retrieval, the citations, the Arabic, is table stakes that a good general model now clears.
The part that went wrong
We should publish this bit, because it is why the top row says coming soon and not available today.
We ran V2 on the same prompt twice, four minutes apart, in the same environment. One run made zero web searches, correctly announced that it could not retrieve the Lebanese legislation, cited no articles, and scored 66. The next run made thirty web searches, cited eighteen articles, every one of them accurate, and scored 88.
Same question. Same build. A twenty-two point swing decided by whether a tool happened to fire, with nothing in the interface to tell the two apart except a paragraph of caveats most readers would skim.
There is a version of this post that leaves that out and reports 88. We think the number is worth less than the disclosure. A legal AI that is excellent half the time and honest about it the other half is a research prototype, not a product, and we would rather say so than ship the headline.
Two earlier attempts also failed outright before generating anything. Those are filed and being fixed.
What we would tell the client
Not the arrangement as described. The income structure and the unpaid family administration can both be drafted. The ending cannot be promised: Article 10 fixes the destination, Article 27 takes its fifteen percent, and Articles 37 and 38 protect the other heirs in anything above a third of the estate.
The correct advice is to settle where the property is legally allowed to end up before the endowment is created, not to create it and try to get the property out afterwards. Every system that scored above 70 reached some version of that. Most of the field did not.
Key takeaways
- Multiple-choice benchmarks do not predict behaviour on a real client instruction. Test the instruction.
- Score against the primary text. Models agree with each other more than they agree with the statute.
- The gap between a frontier general model and a legal product is now one point on knowledge and a wide margin on whether the system is willing to tell the client no.
- A tool that fires intermittently is worse than a tool that is absent, because the output looks identical either way.
- Lebanese family endowment law is narrow, mandatory and under-served. The same is true of most MENA personal status law, which is exactly why we build for it.
- Law of 10 March 1947 on family endowments, full text, Lebanese University Legal Informatics Centre
- ArabLegalEval: a multitask benchmark for Arabic legal knowledge
- IslamicLegalBench: evaluating LLMs across Islamic legal traditions
- Benchmarking LLM reasoning in Arabic Islamic inheritance cases
Single run, one matter, September 2026. We are replicating across more MENA matters and will publish the re-run whether or not it agrees with this one.
