What Happens Before a Legal AI Answers Your Question

Many legal AI products pass your prompt straight to a general model. What a system does before that call shapes the answer. The architecture, and four demo tests.

Key facts

• Language models use long inputs unevenly: accuracy is highest when the relevant information sits at the start or end of the context and drops when it sits in the middle (Liu et al., Lost in the Middle, TACL vol. 12, 2024).
• All 18 models in Chroma's Context Rot study, including GPT-4.1, Claude 4 and Gemini 2.5, got less reliable as input length grew, even on deliberately simple tasks (Chroma technical report, 14 Jul 2025).
• Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated on 17% to 33% of test queries in a preregistered Stanford study (Magesh et al., Journal of Empirical Legal Studies, 2025).
• In our June 2026 run, 24% of 3,000 graded answers from 10 frontier models cited or applied law that did not say what the model claimed (HAQQ benchmark, 5 Jun 2026).
• Damien Charlotin's database listed 2,202 court and tribunal decisions involving AI-fabricated citations or content as of 19 October 2026.

Every lawyer evaluating legal AI eventually asks the same question, usually about eight minutes into a demo: how is this different from using ChatGPT?

Vendors tend to answer with training data, a proprietary corpus or a fine-tune. Those claims are hard to check and often sit away from where the difference lives. The answer we give is architectural, and it comes down to where the thinking happens.

Is most legal AI a ChatGPT wrapper?

A large share of legal AI products work like this. You type a question. The product wraps it in a prompt template (instructions to act as a helpful legal assistant, perhaps a few examples) and forwards it to a general-purpose model. The model answers, and the product formats the answer.

That can be useful. It also means the product's quality is mostly the model's quality, and your opposing counsel can buy the same model on a consumer subscription.

For a buyer, the hard part is that you cannot tell from the interface. A wrapper and a purpose-built engine both show a text box, a streaming response and some citations underneath. The difference sits upstream, out of sight, which is why vendors are rarely pressed on it.

What should happen before the model is called?

When a question reaches HAQQ, it first goes through a step we own, whose job is to work out what the request is and to configure everything that follows.

That step settles what a competent associate would establish before starting work, and it records the result as structured data:

• The question underneath the words. What someone types at 11pm between two calls, restated precisely and completely. This runs natively in Arabic and French as well as English, with no translation step added afterwards.
• The kind of legal work. Drafting, research, review, comparison or a procedural question. Each needs a different shape of answer, and deciding that up front beats hoping a general model infers it.
• The jurisdiction, and separately the language the governing law is written in. The two often differ, and treating them as one is a quiet, common source of error.
• The facts, as objects. Parties, dates, obligations, amounts, governing law and document type, pulled out of the prose before anything expensive reads it.
• What the question needs. Which specialised capabilities to load, how much reasoning depth it warrants, and whether it is in scope at all.

The general-purpose model runs after all of that, and it starts prepared: the relevant capabilities loaded, the facts extracted, the jurisdiction identified and the scope decided.

The model runs last, on work that has already been sorted.

Why does that produce better answers?

Two mechanisms, and the second one surprised us more.

Attention is a budget

A model's context window is finite, and everything in it competes for attention. The research above shows what happens as context grows: retrieval gets worse in the middle of long inputs (Liu et al.) and reliability falls with length even on easy tasks (Chroma). Loading every capability a system has into every request spends that budget before the lawyer's document arrives. Loading only the relevant ones leaves room for the contract, the statute and the facts.

Structured facts spare the model a parsing job

This is the larger effect. Handed a paragraph of prose, a model spends part of its effort working out what the paragraph contains. Handed structured facts (these parties, this governing law, this obligation, this date), it can spend that effort on the legal question.

That is the effect we design for: the same general-purpose model doing better legal work because of what it was handed. We have not published a controlled comparison of the same model inside and outside our engine, so treat this as our design rationale, supported by the context research above.

What decides whether legal AI works on real documents?

Two components that rarely appear in a sales deck, and both decide whether the product works on a normal Tuesday.

Document intake. Much legal work arrives as a photograph of a stamped page, a scan of a fax or a PDF that was printed and re-scanned crooked. We treat reading those as a core engineering problem. If a clause number is misread on the way in, no downstream reasoning recovers it, and the system will reason carefully about the wrong number.

Memory as structure. The engine keeps the relationships between your matters over time and draws on them when a question calls for it. Pasting your last twenty messages back into the prompt is a different thing. Structured memory is what lets the system know that the counterparty in the NDA you are reviewing today is the same company from a dispute two months ago.

What does good architecture not fix?

We would rather say this than have you find it out.

It does not eliminate error. Any system built on a language model can produce a wrong answer, and the figures in the key facts above apply to every tier of the market. Good architecture makes errors rarer and easier to catch: claims generated against retrieved sources, citations you can open, and explicit flags where the law is unsettled. Our hallucination audit tracks what happens when nobody checks.

It does not remove the lawyer. Everything here assumes a professional reviews the output and signs it. That assumption shaped the architecture from the start.

It does not make coverage universal. Quality varies by jurisdiction, by how well the source law is digitised and by how much of it is public. If a vendor quotes a single global coverage number, ask what it counts. We would rather answer that question about your jurisdiction than give you one figure.

How do you test a legal AI vendor in a demo?

You do not need anyone's architecture diagram. Four tests, about ten minutes, and they work on every vendor in the category, us included. For a longer checklist, see 45 red flags in a legal AI vendor evaluation.

1. Ask it something outside law

A general model in a legal prompt will usually answer a medical or financial question, because underneath it is a general model. A system that establishes intent before generating should decline and say why.

2. Ask the same question in two languages

Write the same legal question once in English and once in Arabic or French, each in your own words. If the substance of the answer changes, language is being handled after the reasoning.

3. Give it a bad scan

Photograph a document at an angle, in poor light, and upload it. In our experience this is the most predictive test in the demo, because it is the most common real input and the least demoed one.

4. Ask what it does not know

Push it toward an unsettled or thinly sourced area of law. A system tuned to always produce an answer will produce one. A system built for professional work will tell you the ground is uncertain and show you why.

If a vendor is comfortable with all four, the architecture is probably real. If the demo steers around them, that tells you something too. You can run the same four against HAQQ's legal AI chat.

Key takeaways

• The meaningful difference between legal AI products is architectural and invisible from the interface. Ask where the reasoning happens.
• Structured work before the model call improves quality as well as cost: a model handed extracted facts spends its attention on law.
• Unglamorous components decide real-world performance. Document intake predicts whether a tool works on a Tuesday better than a benchmark score does.
• Four demo tests (off-topic, two languages, a bad scan, unsettled law) show what kind of system you are being sold.

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
• Chroma: Context Rot, how increasing input tokens impacts LLM performance (Jul 2025)
• Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (JELS 2025)

• 45 red flags when evaluating a legal AI vendor
• Context engineering for legal AI
• Human in the loop legal AI
• The AI hallucination audit