We ran GPT-6 Astra against 41 real legal questions

We ran GPT-6 Astra against 41 real legal questions, alongside Claude Opus 5 and GPT-5.6 Luna Pro. Every model we tested got Saudi and Egyptian law wrong while citing the correct article — the failure no citation checker catches.

What we ran

41 prompts across 20 practice areas, from M&A and securities to immigration and construction. Three models: GPT-6 Astra, Claude Opus 5, GPT-5.6 Luna Pro. Reasoning effort pinned to medium for all three, so the comparison is fair rather than flattering. Nine properties measured separately, because a model that drafts beautifully and invents a statute is not a good legal model. $66.55 of metered inference. Run 10 September 2026.

HAQQ is not in this benchmark. We did not enter our own product into a study we ran. What follows is three frontier models, measured against each other.

The leaderboard is close. The price is not.

On legal substance — judged quality plus accuracy — Astra leads at 17.63 of 20. On the composite, which folds in a speed rank, Luna Pro takes it at 30.93 of 35, against Astra's 30.05 and Opus 5's 28.13.

Both are true, which is the point. A benchmark that reports only the composite is quietly telling you latency matters as much as being right.

Nobody buys a token

Astra bills twice Opus 5's per-token rate and still costs 1.7x less per answer, because it writes 3.5x fewer tokens to say the same thing.

The nominally cheaper model is the more expensive one.

Terseness is the finding. Astra's advantage is not that it is smarter, it is that it stops writing.

Against Luna Pro the trade collapses: 11x the cost per answer for under a point of composite. For general legal Q&A at volume, that is not defensible.

The reasoning dial is a cost lever, not a quality lever

Astra exposes a reasoning-effort setting, so we ran ten prompts at each level. Low to high multiplied cost by 1.5x and latency by 1.8x, and moved judged quality by -0.17 points.

Turning it up made answers slightly worse and meaningfully more expensive. If you run Astra, pin the effort.

The part that should worry you

Six questions across five jurisdictions, asked in the language a local practitioner would use. Saudi labour notice periods and UAE probation limits in Arabic. French limitation periods and unfair-terms doctrine. Qatari transfer rules. Egyptian data law.

Astra: 66.7% legally correct. Luna Pro: 66.7%. Claude Opus 5: 100%.

All three replied in the right language, every time. All three cited the right provision, every time. The two OpenAI models then misstated what that provision says — Saudi Labour Law Article 75 notice periods, Egyptian PDPL cross-border consent.

A citation checker passes that answer. Every hallucination-detection tool on the market passes that answer. A client does not.

This is the gap between fluency in a language and correctness in a jurisdiction, and nothing about a bigger context window or a better reasoner closes it.

Long context is solved. Paying for it is not.

We planted four conflicting clauses in a Master Services Agreement — a cap, a carve-out that overrides it, a compounding escalator, and a governing-law clause — at 6%, 24%, 47% and 68% of its length. Answering correctly means finding all four and applying the right precedence. Finding only the first produces a confident, specific, wrong number.

All three got it right at 25k, 100k and 200k tokens. But reaching the same answer at 200k cost $2.01 on Astra and $0.15 on Luna Pro. A 13x spread for identical output.

Determinism is not available at any price

Same question, same seed, five times. Not one model reproduced an identical answer. Word-level similarity ran 0.47 to 0.78. Two of the three accept a seed and still do not produce the same text twice.

If the same contract question returns a different answer on Tuesday, it was never reliable on Monday. Anything you promise a client about consistency has to be built above the model, not bought from it.

What we would tell a firm choosing today

Route by matter type, not by brand. Our per-practice-area table shows Astra ahead in most areas and Opus 5 ahead in insolvency and ESG, with gaps small enough that the decision is usually cost and latency rather than capability.

Do not move high-volume legal Q&A to Astra expecting a quality jump. Do consider it where you are paying Opus 5 for long answers nobody reads.

And whichever you pick, verify the jurisdiction. All three will hand you the right article number attached to the wrong rule, in the languages our market works in.

Limitations, stated plainly

This rests on fewer prompts than we planned, and Claude Opus 5 answered 37 of the 41 against 41 for the other two, so the base is uneven. Treat the leaderboard as provisional and the multilingual result as the finding worth acting on. We will re-measure when the full set is complete and publish what it says.

We ran this ourselves. It is not independently audited. The method is above so you can check it.

• The full study on HAQQ Academy