Best AI for Legal Research? 3 Models vs 100 Real Questions (May 2026 Run)
Our May 2026 run: Claude Sonnet 4, GPT-4o and Gemini 2.5 Flash on 100 real r/legaladvice questions. Pass rates 78 to 88%, caveats the weak spot. Newer runs linked.
Key facts
- Pass rates on 100 all-time top r/legaladvice questions: Claude Sonnet 4 88/100, GPT-4o 87/100, Gemini 2.5 Flash 78/100 (HAQQ run, May 2026).
- Weakest dimension for all three: Appropriate Caveats, 3.0 to 3.15 out of 5. Legal Accuracy ranged from 3.98 to 4.30 (HAQQ run, May 2026).
- On 20 fresh questions posted in the previous 48 hours: Claude Sonnet 4 95%, GPT-4o 90%, Gemini 2.5 Flash 85% (HAQQ run, May 2026).
- Judge: Claude Sonnet 4 graded every answer, including its own, with no attorney validation (HAQQ run, May 2026).
- Context: the World Justice Project estimated in 2019 that 5.1 billion people have unmet justice needs (WJP, Measuring the Justice Gap, 2019).
How did we set up the test?
We took the 100 highest-voted posts of all time on r/legaladvice: real questions about landlord and tenant disputes, employment, custody, criminal charges, personal injury and more. The average post ran past 2,200 characters, so most questions came with messy facts attached.
Each question went to three models through OpenRouter with the same system prompt: act as an experienced US attorney, identify the jurisdiction, spot the issues, cite the applicable law, analyse, then advise.
- Claude Sonnet 4 (Anthropic)
- GPT-4o (OpenAI)
- Gemini 2.5 Flash (Google)
How were the answers graded?
Claude Sonnet 4 acted as the judge and scored each answer from 1 to 5 on five dimensions: Legal Accuracy (are the cited laws correct?), Issue Completeness (did it catch every legal issue?), Reasoning Quality (does the reasoning hold together?), Practical Value (would it help the person take the right next step?) and Appropriate Caveats (does it say it is not legal advice and point to a local attorney?).
An answer passed if its average was at least 3.5 and no dimension fell below 2. Using a model to grade models brings bias, which we come back to below.
Which model passed the most questions?
Claude Sonnet 4 passed 88 of 100, GPT-4o 87 and Gemini 2.5 Flash 78. With 100 questions and a single judge, a one-point gap between Claude and GPT-4o is noise: treat them as tied.
How did the models score on each dimension?
Legal Accuracy ran from 3.98 to 4.30 out of 5. Gemini scored highest on Issue Completeness (4.82), ahead of Claude (4.58). Claude's strongest dimension was Practical Value at 4.73, against 4.21 for GPT-4o, mostly because its answers listed more concrete next steps. Caveats were the low point for every model.
What did we learn in May 2026?
The models usually found the right area of law
In most of the 100 cases, each model identified the area of law, spotted the main issues and gave advice a person could act on. That was the encouraging part of the run.
The weak point was caveats
All three models scored 3.0 to 3.15 on Appropriate Caveats. They would go deep into the analysis, often correctly, without saying that this was general information or that the person should see a local lawyer. A confident answer with no warning is risky even when it is right, because the reader cannot tell which parts depend on their state or their facts. This is one reason we put guardrails, disclaimers and escalation to a lawyer on top of any model HAQQ uses.
High averages did not mean a high pass rate
Gemini 2.5 Flash had the highest Legal Accuracy (4.30) and Issue Completeness (4.82) averages but the lowest pass rate, 78%. Some of its answers were cut off; others skipped the disclaimer entirely and failed on that dimension. For legal work, an answer that is excellent most of the time and incomplete the rest is hard to rely on.
How reliable is an AI judge?
Claude judged all three models, including itself. That raises three problems we could not remove in this run: a possible home-court advantage, a bias toward answer structures the judge recognises, and no ground truth, because no attorney checked the answers. What the scores measure is agreement with one model's view of a good answer.
Our later runs addressed this. The September 2026 Opus 5.5 benchmark used two non-Anthropic judges, and the trap-question test used three blind judges and published all 80 answers.
Did the results hold on fresh questions?
The top-100 posts are old enough to have appeared in training data. To check for that, we ran the same pipeline on 20 questions posted to r/legaladvice in the previous 48 hours. Claude Sonnet 4 scored 95%, GPT-4o 90% and Gemini 2.5 Flash 85%. Twenty questions is a small sample, so we read this as "no sign of memorisation" rather than as an improvement.
What does this mean for a legal AI product?
Even in May 2026, the models could reason about common US legal questions well enough to be useful most of the time. The remaining 5 to 15% is where a product earns trust: checking which model's answer to keep, adding the caveats the models skipped, grounding answers in the firm's jurisdiction and documents, and making every citation traceable to its source. That is the work Justinian, HAQQ's legal AI engine, does on top of the models it runs.
Methodology
- Run date: May 2026.
- Questions: the 100 top posts of all time on r/legaladvice, plus 20 posts from the previous 48 hours, filtered to substantive text posts.
- Models: Claude Sonnet 4, GPT-4o and Gemini 2.5 Flash, all called through OpenRouter with identical system prompts.
- Prompting: chain of thought with a five-step legal reasoning framework (jurisdiction, issues, law, analysis, advice).
- Judge: Claude Sonnet 4, single judge, five dimensions scored 1 to 5, no attorney review.
- Pass rule: mean of at least 3.5 and no dimension below 2.
- Data: code, questions, answers and scores were kept in a repository at the time of the run.
What should you read next?
- Best AI for legal work, updated October 2026
- Claude Opus 5.5, Opus 5 and Fable 5.1 on ten legal prompts
- How to test a legal AI: five trap questions
- Where AI hallucinations have reached a court
- World Justice Project, Measuring the Justice Gap (2019)
FAQ
Which AI did best on 100 real legal questions in May 2026?
In our May 2026 run on the 100 top r/legaladvice questions of all time, Claude Sonnet 4 passed 88, GPT-4o 87 and Gemini 2.5 Flash 78. The one-point gap between Claude and GPT-4o is within noise for a single-judge test of this size. These models have since been replaced; see our October 2026 best-AI-for-legal-work page for current results.
What was the weakest area for AI legal answers?
Caveats. All three models scored 3.0 to 3.15 out of 5 on Appropriate Caveats, against 3.98 to 4.30 on Legal Accuracy. They often gave detailed and correct analysis without saying it was general information or recommending a local attorney.
How was the May 2026 benchmark scored?
Claude Sonnet 4 graded every answer from 1 to 5 on Legal Accuracy, Issue Completeness, Reasoning Quality, Practical Value and Appropriate Caveats. An answer passed with a mean of at least 3.5 and no dimension below 2. There was no attorney review, so the scores measure agreement with one model's judgement.
Is this benchmark still current?
No. It is kept as a historical record of a May 2026 run on Claude Sonnet 4, GPT-4o and Gemini 2.5 Flash. Newer HAQQ runs from September 2026 test Claude Opus 5.5, GPT-6 Astra, Gemini 3.1 Pro and DeepSeek V4 Pro with multiple blind judges and published answers.
Is ChatGPT good enough for legal research?
General chat models can explain common legal questions in plain language, and GPT-4o passed 87 of 100 questions in our May 2026 run. They can still invent citations, miss jurisdiction-specific rules and skip warnings, so treat their answers as a first draft and verify every authority against a primary source before relying on it.