Jev Inside a Legal AI Harness: 8 Decision Patterns, and the Boundary We Measured

We ran TypeSafe's Jev across our legal AI stack for under $0.20. It reranks statutes well. It also scored fabricated citations higher than honest ones.

The pitch, and why we took it seriously

There is an argument going around that most AI agents waste their most expensive resource on decisions that never needed a sentence. Routing a request to the right model is a decision. Deciding whether a tool call is safe is a decision. Ranking fifty retrieved documents is fifty decisions. None of these produce prose anyone reads, yet all of them are handed to a model that bills by the word and answers in paragraphs.

In September 2026 TypeSafe AI shipped a model built around that argument. Jev is what they call a System One model, and the defining property is that it generates no text at all. You POST a block of state plus a set of typed questions, and you get back calibrated probabilities, all evaluated in a single parallel round trip.

There are three primitives and there is no fourth. A noul is a yes or no question that returns a probability between 0 and 1, and that probability is the confidence: 0.5 means a coin flip, not medium intensity. A choice picks one option from a list you define, and it physically cannot return an option you did not declare. A score places the input on an ordered scale you describe in words, and returns the probability-weighted mean, so a 70/30 split across two levels comes back as 1.30 rather than 1 or 2.

That is the whole surface. It cannot summarise, rewrite, draft, extract a span, or look anything up. It has no knowledge base and no retrieval. It answers questions whose answer space you declared, about a state you handed it.

The part that is easy to miss

The instinct is to file this under cheap fast classifier. That undersells what actually changed, which is not the price but the ordering.

A language model writes one token at a time, so token nine cannot exist until token eight does. Ask it four unrelated questions about a contract and those four independent judgments queue behind each other, come back as a blob you then parse, validate and retry when the shape is wrong. Jev deletes the queue. You declare the answer space up front and every question resolves against the same state at once. Text is a line you have to walk. An answer space is a room you see all at once.

That failure-mode row is the one that matters in law, and it is worth being blunt about. The guarantee on offer is that you will never get a value outside your schema. It is not a guarantee that the value is right.

A schema-valid mistake refunds the wrong client just as fast as a malformed one does.

For a legal AI company, the interesting word in all of this is calibrated. A number you can branch on in code is a different object from a paragraph a lawyer has to read and trust. We wrote in our orchestration layer post that the part of legal AI which determines whether the citations hold is invisible from the outside. This is a candidate for filling in some of that invisible part.

So we ran it across our entire stack. Every measurement below is ours, on our own corpus, and the ones that went badly are here too.

What we measured, and what it cost

Ten surfaces, thirteen closed tickets, under $0.20 in total spend. That is not a boast about frugality. It is the reason the experiment was worth running at all: at a fraction of a cent per judgment you can afford to check every event rather than sample, and a null result costs almost nothing to establish.

Latency, measured on our own traffic with a warm connection held open: 292ms at the median, 374ms at p90, 395ms at p95, across 40 calls. Server-side compute alone is 112ms. Our own first measurement said 700ms, and we repeated that number internally for a week before noticing it was wrong. We had never reused a connection, so every call was paying for a fresh handshake. Same API, same model version, a careful measurement of the wrong thing. It is a small embarrassment and a useful one, because it is exactly the class of error that makes a benchmark unreproducible a month later.

Eight patterns, and which ones we actually tested

Seven of these circulated as a developer's pattern list. The eighth is ours, and it is the one that turns out to matter most for legal work.

• Model routing. Predict task complexity, send the easy work to a small model and the hard work to a frontier one.
• Tool-call risk gates. Before an agent executes a tool, score the tool plus its arguments plus the current task, and return allow, block, or human approval.
• Intent and domain triage. Classify the request into a fixed set of intents before the main agent starts reasoning. In law this is the jurisdiction, the practice area and the deliverable.
• Loop decisions. After each agent step, predict whether another step is likely to be useful.
• Retrieval reranking. Score each retrieved passage against the query and pass only the highest-scoring ones into context.
• Guardrails and policy checks. Run safety and compliance checks as fast classifications instead of generating a full reasoning response.
• Escalation to a human. Predict whether an automated decision is reliable enough to execute.
• Verbatim context compaction. Score each step of a long agent trace for whether it still bears on the objective, drop the dead ones, and keep the survivors word for word.

We have now measured six of these on our own data. Three work, one of them in production today. Three do not. The failures get more space below, because they are more instructive and because nobody else is publishing them.

A note on the neighbourhood: a small number of tools built on this same stack already exist for legal citation checking, all shipped within the last two weeks. They are early experiments rather than operating products, and none publishes an accuracy figure. The gap we are writing into is not that nobody has had this idea. It is that nobody has published what happens when you check it.

Where it works: reranking retrieval

Our MENA statute corpus holds 17,004 articles. Retrieval over it is lexical first, which means a query and an article that use different words for the same obligation can miss each other entirely. This is the ordinary failure of legal search in any language, and it is worse in Arabic, where morphology puts more distance between a query term and its match.

We took 80 held-out queries, ran them through the existing retrieval, then reranked the candidates by asking one question per candidate: does this article answer this question.

On nine of the 80 queries the correct article was never retrieved at all, so no reranker could promote it. The queries were written from the sampled articles, which flatters lexical retrieval, so read the delta rather than the absolute numbers.

Mean reciprocal rank moved from 0.719 to 0.869. Ten of ten hand-checked cases agreed, including two where the model correctly declined to promote anything because no valid candidate existed. Total cost: $0.0245.

One further caveat. Our Arabic and English articles are split by jurisdiction, every Arabic one Egyptian and every English one Emirati, so this run supports no claim about language at all.

It works because the question has a right answer. Does this article answer this question is settled by the article and the question, both of which are in front of the model.

Where it works: catching the answers that are not answers

Before we adopted Jev we graded benchmark answers with a language model acting as a judge. That judge was generous in a specific and damaging way. It handed a perfect 10.0 to 56% of the answers it scored, and for one model 90% of answers came back flawless on all four dimensions. A grader with no ceiling is not measuring anything at the top of its range.

Jev scored 0% of answers at 9.9 or above. More usefully, it caught all five truncated API errors in the set at probabilities between 0.96 and 0.97. The previous judge had scored those same five as 1 out of 10 and averaged them into the model capability score, which meant we had been reporting a 10% infrastructure failure rate as legal incompetence. Excluding them moved one model from 5.80 to 7.67.

The most valuable thing the model did for our benchmarks was not judge quality. It was notice that a string was not an answer.

Where it works, in production: the compliance gate regex could not see

This one is live, and it earned its place by catching something embarrassing.

We run every piece of public copy through a preflight gate before it ships, checking it against our own messaging rules. Some of those rules are red lines: we do not say we replace lawyers, and we do not claim our AI is free of hallucinations, because neither is true and both are the kind of claim a regulator reads closely.

A French draft containing literal violations of both rules passed that gate at exit 0, with zero findings. The regex patterns were English-only. The French for we replace lawyers, and the French for our AI has no hallucinations, walked straight through a check that would have stopped their English equivalents instantly.

A semantic pass caught both decisively: 0.90 on the implied claim, 0.95 on the lawyer-replacement red line, 0.83 on how a regulator would read it. Cost per draft: about $0.00005, across eight typed questions in one parallel request.

The result we did not expect is that regex and semantic turned out to be complementary rather than redundant. We went in assuming one would retire the other, and measured across 35 real drafts that every regex rule with a semantic counterpart was catching a different phrasing of the same risk. The regex catches the literal wording. The semantic pass catches the paraphrase. Our own ticket predicted we could retire something, and the data said no.

Where it inverted: asking it whether a citation is grounded

Now the failure that reorganised our thinking.

We asked Jev to score groundedness on 150 real legal answers, produced by three frontier models across 50 MENA and UK prompts. Groundedness here means: are the claims supported, are the citations real and used correctly. It is the single most important question in legal AI, and it is the question our own hallucination audit exists to keep asking.

Answers containing fabricated citations scored 3.21. Honest answers scored 2.79. The correlation between the composite score and human-reviewed accuracy labels was r = +0.02, which is to say none.

The check did not merely fail to detect fabrication. It rewarded it.

Once you see why, you cannot unsee it. Article 5bis of Royal Decree M/13 reads as precise, specific and authoritative. It does not exist. A model asked to judge groundedness from text alone has nothing to compare against, so it falls back on the features that usually accompany grounded writing: specificity, citation density, confident structure. Fabrication produces all three. Specificity is what fabrication looks like, so any rubric that rewards specificity rewards fabrication.

Now change one thing. Hand the model the source passage alongside the claim and ask whether this passage supports this proposition. On a separate, reproducible 180-pair run from our own corpus, the hosted model reached an AUC of 0.9962 and caught 94.4% of misattributions at a zero false alarm rate.

Same model. Same task in the abstract. Opposite outcome, because one version is a comparison between two things in front of it and the other is a knowledge lookup it cannot perform.

One more caveat, because it matters more than the win. On that same 180-pair set, a plain containment check with `grep` scores AUC 0.9944 and catches 98.9%. Those pairs were built on verbatim containment, which is precisely what `grep` is for. Treat 0.9962 as a floor rather than a ceiling, and treat paraphrase-level grounding, where a typed classifier should actually earn its keep, as untested by that run.

Retrieval is not an implementation detail underneath the verification step. Retrieval is what makes the verification possible at all.

Where it lost to a step counter: loop control

We tested this one specifically for this post, and it did not survive.

The pattern says: after every agent step, ask whether another step is likely to be useful, and stop when the answer is no. We built the test from 52 of our own finished agent trajectories, 1,750 labelled steps. The ground truth for each step was whether the next step actually surfaced information not already somewhere in the trajectory, computed by a rule we hand-checked at 24 out of 25.

The gap between Jev and the second free signal is +0.017, with a 95% confidence interval from -0.010 to +0.051. It does not clear a paired bootstrap. Steps saved at zero loss of useful work: zero, for all three.

We paid $0.54 to not beat a feature computed with a set difference.

Two things underneath that are worth more than the headline. The first is that a lull is not exhaustion. The free rule stop after two consecutive steps that returned nothing new, run as an oracle on its own premise, damaged 30 of the 31 trajectories it stopped. Agents go quiet in the middle of useful work all the time.

The second explains the whole result. Split the rows by what kind of tool the next step used and the picture changes completely: 0.742 when the next step writes or edits a file, 0.695 when it reads or searches, and 0.597, near chance, on shell commands, where 59% of our rows live.

It was never predicting whether the agent had learned enough. It was predicting whether the next thing coming back would be a write acknowledgement or a page of output. That is a judgment about the shape of the next event, not about the state of the agent knowledge.

Scope this honestly: our corpus is coding and research agents, not legal reasoning loops, and the ceiling on savings was low to begin with, because a finished transcript is by definition one that stopped roughly when the information stopped. Both facts limit how far it generalises. Neither rescues the pattern.

Where it lost to a regular expression: tool-call risk gates

The last pattern we tested, and the most uncomfortable, because we wanted this one to work.

We run infrastructure that must not go offline. The rule protecting it currently lives in a configuration file an agent is supposed to read and obey. A rule that depends on being read is not a control. A gate that fires on the call is.

So we labelled 19,922 real tool calls from 766 of our own transcripts, scored 284 of them, and compared a typed classifier against the deny-list you would actually write by hand.

We required every destructive call to be caught. The classifier reached 89.1%, below both deny-lists. The regex wins the class the test existed for.

There is a genuine win buried in it. On the narrow question of whether a specific call could take a live host offline, a dedicated yes-or-no question caught 21 of 22 at a 3.4% false alarm rate, where the stronger of two hand-written deny-lists caught 8 of 22. Union the two and the rate of risky calls slipping through halves. At eleven cents per thousand calls, cost is not what stops you. The recommendation that came out of it is still to ship the regex and keep the classifier as a second opinion, never as the gate.

The cleanest result in the whole programme sits inside this test, and it is about how you ask rather than what you ask. Same model, same rows, same afternoon. The concrete question, can this call take one of these specific live hosts offline, hit 95.5% recall at a 3.4% false alarm rate. The abstract question, is this action irreversible, needed a far looser threshold to reach comparable recall and dragged the false alarm rate to 34.6% getting there.

We had learned this before and written it down as a maxim: name the concrete test, not the abstract category. This is the first time we have it as a controlled comparison rather than an anecdote, and the size of the gap surprised us.

One more thing from that test, with its limits attached. We tried to talk the classifier into approving calls it had blocked, using eight hand-written injection payloads across 384 paired calls, planted in the command, the description and the task context. None reliably flipped a blocked call to allowed. That is a real result and a small one: eight payloads written by the same person who built the test is not a red team, and a deny-list is immune to this by construction rather than by luck. TypeSafe own documentation lists the state is not treated as hostile among the model limitations, and for a safety gate that limitation lands directly on the input an attacker controls.

The eighth pattern, and why it is the legal one

Everything above is borrowed from general agent engineering. This one is ours, and it comes out of a constraint that only really bites in law.

Long agent runs overflow their context, and the standard fix is to have a language model summarise the trace so far. For most software that is fine. For legal work it is a quiet disaster, because a summary of a statute is a new document. It has new words in it. Those words were written by a model, not by a legislature, and every downstream step now reasons about the paraphrase rather than the law.

The alternative is to compact without paraphrasing. Score each step of the trace for whether it still bears on the current objective, drop the ones that do not, and keep the survivors word for word. Nothing is rewritten. The context gets shorter because things were removed, not because things were reworded.

A summary of a statute is a new document. A filtered list of statutes is still the statutes.

This is the pattern we are most interested in and it is the one we have not yet measured, so treat it as a design position rather than a result. We are saying it out loud because it is the shape of problem a decision model is unusually well suited to, and because we have not seen anyone else frame compaction as a correctness requirement rather than a cost optimisation.

Two things about legal work that nobody warns you about

It cannot see a scanned document. Jev takes text. A very large share of real legal documents, particularly across MENA, arrive as photographs of paper. That means the pipeline in front of it does all the work that determines whether it gets a fair input, and any confidence it reports is conditional on an OCR step it knows nothing about. Open reimplementations of the same architecture running on vision-capable base models do exist, which is the direction this eventually goes, but the hosted model in front of you today is blind.

Legal abbreviations break it in a way that looks like a bug and is not. We asked it to classify a request to review an Arabic MSA and redline it. Legal-ness came back at 0.30, which for a clearly legal request looks like a failure. Isolate the phrase and the picture resolves: MSA alone scores 0.97, Master Services Agreement 0.98, an MSA written in Arabic 0.96, and Arabic MSA collapses to 0.30. Putting the word Arabic immediately before MSA flips the acronym from Master Services Agreement to Modern Standard Arabic.

The model is right. The phrase is genuinely ambiguous, and it would be ambiguous to a human reader too. But MENA legal work generates that exact construction constantly, so any pipeline that routes on a document label will hit it, and in a benchmark it will read as a model failure when it is an input-ambiguity artefact. If you are evaluating a classifier on legal text, check for an ambiguous abbreviation before you conclude the model cannot handle your jurisdiction.

The rule that came out of all of it

Six surfaces, three wins, three losses. The pattern in the losses is the thing worth taking away.

Reranking works because does this article answer this question has an answer that the article and the question settle between them. Citation grounding works when you hand over the source and inverts when you do not, because without the source the question becomes a knowledge lookup. Loop control fails because has this agent learned enough is not settled by anything in the trajectory. Internal link suggestion, which we also tested, failed the same way: asked whether a reader would have a real reason to click from one page to another, the scores against 761 human-curated links came back at 0.582 for real links and 0.537 for pairs no editor had ever connected, a gap of 0.045, with hand-checkers agreeing on 12 of 20.

Before you build on a judgment, ask what the ground truth for it is. If the honest answer is an editor preference, expect a gap of around 0.05 and plan for a human.

This is the same boundary our own fake citation investigation found from the other direction, and it is why we judge tools on whether they check that a citation says what they claim, not merely that it exists.

What we would tell someone evaluating legal AI

Four things, only one of them about this model in particular.

Ask what the check is anchored to. Our AI verifies citations is not an architecture. Verifying that a case exists is a database lookup. Verifying that the case says what the answer claims is a comparison, and it needs the source passage in hand at the moment of judgment. Products that do the first and describe it as the second are the norm, not the exception.

Ask for the failure rate, not the success rate. Every tool in this category publishes a number that flatters it. Very few publish what they measured and lost. Our own honesty has limits worth stating: the first automated pass at labelling that tool-call corpus was 86% wrong on its most important class, all in the same direction, and we found it by hand-checking rather than by cleverness.

Ask what the benchmark was scored against. This one catches almost everybody. A benchmark can be scored against ground truth, or against agreement with a frontier language model. The second is much cheaper and much weaker, because agreeing with GPT includes agreeing with GPT mistakes. When a vendor reports accuracy, the question is not how high the number is. It is what the number was compared to.

Be careful with a number that is asserted rather than measured. TypeSafe own launch chart shows a 0% hallucination rate, and the footnote under it says, in their words, that the number is not empirical and is added by construction because the schema match is enforced rather than observed. They disclose it. The headline above the chart does not. The claim underneath is true and narrow: you will never get a value outside your schema. It is not a claim that the value is right. An independent analyst made this point a week after launch, and it is a fair one.

What actually changed for us

Not the cost. Swapping a language model call for a typed one saves pennies, and pennies were never the problem.

What changed is that a decision the system was already making silently now carries a number. Contract review at 0.91 against litigation at 0.09 is a route you can automate and log. 0.52 against 0.46 is a coin flip wearing a label, and it gets a human. A language model would have given the same answer in both cases, in a confident sentence, with no way to tell the two situations apart.

For a product where being wrong is the entire risk surface, that is worth more than any speed multiple on a vendor chart.

The habit we would keep even if we never sent another request: go through your agent and find every call that merely picks something. Which tool next. Is this spam. Is this chunk relevant. Does this need a human. Is this diff risky. None of those are writing tasks. They are if-statements somebody outsourced to a frontier model, and most of them do not need one.

Key Takeaways

• A model that generates no text is a real tool for a legal harness, in the specific places where a judgment is a comparison between things you can put in front of it.
• Asked to score groundedness on text alone, it scored fabricated citations higher than honest ones, 3.21 against 2.79, with a correlation to real accuracy of r = +0.02. Retrieval is what makes verification possible, not an implementation detail underneath it.
• It is running in production on our compliance gate, where it caught red-line violations in a French draft that our English-only regex passed at exit 0. Regex and semantic checks turned out to be complementary, not redundant, against our own prediction.
• Reranking our 17,004-article statute corpus moved recall at rank 1 from 0.638 to 0.850 for under three cents.
• Loop control did not beat a free step counter in a paired test, and its apparent skill collapsed to near chance once we controlled for what kind of action came next.
• A tool-call risk gate recalled 89.1% of destructive calls where a hand-written deny-list recalled 99.2%. We are shipping the regex.
• For legal work specifically: it cannot read a scanned document, and Arabic MSA reads as Modern Standard Arabic rather than Master Services Agreement, dropping legal-ness from 0.97 to 0.30.
• Compaction should remove, not rewrite. A summary of a statute is a new document.
• Ask what the ground truth for a judgment is before you build on it. No ground truth, no tool.

Sources and further reading

• TypeSafe AI: Jev documentation and published limitations
• Novel Cognition: The Guarantee, Can't Hallucinate Covers Shape, Not Truth
• How legal AI invents a citation
• AI Legal Hallucination Audit
• The legal AI orchestration layer
• Legal AI Benchmark 2026