Legal work is a good fit for typed decisions because so much of it is reading a document and answering a question whose possible answers are already known. Is this clause capped? Which party bears the obligation? Is this claim likely to be dismissed? The model can answer the judgment; the contract, the escalation and the advice stay with the firm.
Contract review is bounded questions
The Law://WhatsNext episode is a vendor-adjacent demo rather than an independent benchmark. Chris Bridges, a partner and COO at Tacit Legal, describes a simulated contract run in which Jev responded in 0.3 seconds with all of the answers while a chat-based model took 9.6 seconds and returned an unstructured block of text. The structured output went down to whether confidentiality is capped and whether the cap applies to fees paid or payable.
The timing comes from one simulated demo, and the guest works for a firm building supervised AI contract review, so he is not a neutral evaluator. Still, the shape is right: the useful unit is a clause-level question with a closed answer set, not a summary of the contract. The use-case map lists legal and compliance as an example domain for exactly this reason, and the decision-models-as-judges piece covers the evaluation posture such a system needs.
Clause extraction needs a defined answer space
DocJev is an independent document classifier and splitter, not a wrapper around a hosted legal API. It returns one category, probabilities and review flags from a classify call, and ordered categories plus page ranges from a split call. On a 40-document pilot both engines classified 40 of 40 originals correctly, and Jev split 7 of 8 packets exactly against 8 of 8 for a small-LLM baseline. A 15-page public-finance packet kept a ten-page BEA release together and separated two adjacent Treasury results.
Two caveats keep that result honest. A segment's mean category probability is an average of page scores, not a joint probability, and the selected-category probability and provider confidence are different values with neither claimed calibrated. The useful control is a review margin: a boundary score within 0.1 of the threshold flags a cut or continuation. DocJev also still calls the hosted model, so local OCR does not make the pipeline offline.
US Legal MCP takes the opposite approach: it puts legal search into an agent and uses Jev as an optional high-confidence filter. With a key set, query-based tools such as search_congress_bills, search_federal_register and search_court_opinions send each hit to Jev and return only high-confidence matches. Lookup tools are unchanged. The repository publishes no confidence thresholds, latency or pricing, so how aggressively the filter drops hits is not documented.
Stability matters more than a single accuracy number
The ContractNLI study is the most useful independent evaluation here because it measures decision stability, not just accuracy. It evaluates Jev and nine language models with repeated requests that preserve the contract and target while varying visible hypotheses, requested answers and output order. Corrections and regressions can cancel, leaving aggregate accuracy unchanged even though individual decisions moved. Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the error.
The headline result cuts both ways: Jev is the cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy. A four-model follow-up adds an equal-call repetition control and the remaining test contracts, but the finding that consistent answers can preserve errors means aggregate accuracy should not be read as decision stability. If you are reviewing contracts, test the same clause under reordered questions and hypotheses before trusting a threshold.
Forecasting is a different task
Contract review asks what a document says. LegalForecastBench asks what a court will do. It predicts the probability that each challenged claim against each challenged defendant is dismissed in full, using the same written record, and scores the claim-defendant pair with a micro-Brier score and intervals clustered by case family.
The contamination controls are the interesting part: cutoff-eligible decisions, redacted inputs, no network access during official runs, and isolated per-model-per-case GitHub Actions jobs with authenticated reruns. The author states this reduces leakage but does not prove zero contamination. Official model rows are withheld while the full report is assembled, and cohort scores are not comparable across releases. Those are reasons to wait for the report, not reasons to treat the benchmark as invalid.
A forecasting benchmark also forces a distinction the contract-review demos blur: a probability that a claim is dismissed is not a legal recommendation. It is a calibrated forecast over a defined population, and its value depends on the base rate and the case mix.
Triage before review
Claim Triage reads a first-notice-of-loss narrative and a document checklist rather than a transaction ledger. It classifies loss type, scores documentation completeness, and flags suspected fraud indicators in one request. Suspicious claims route to an investigation queue, and the tool never denies or prices a claim. Because Jev is text-only, attachments are reduced to extracted fields and a short checklist rather than passed as images.
The audit trail is the part to copy: every decision stores the narrative state, the question set, the answers, the model version and the thresholds applied. That makes a later review able to reconstruct why a claim was routed the way it was. It is also the minimum for a regulated workflow, where "the model said so" is not a defensible record.
Local models and the limits
Hugging Face's JevK5 is an independent open alternative, not TypeSafe's model or architecture. It reports JevBench v1.2 public scores of 1.000 easy, 0.944 standard and 0.784 hard, with 231/231 valid answers and about 13 ms per decision on an H100 in-process at batch 1. The index proxy is the author's own estimate rather than an index score, and each of its 16 sources has only 40 rows, so any single source is noisy at about plus or minus 0.15.
For a firm, the more important question is the data path. TypeSafe's legal page commits not to train on user data and offers zero data retention for enterprise through sales, but the page is a thin index and the actual terms live off-site, with zero retention enterprise-gated rather than default. That is a contracting question, not a model-quality question, and it should be settled before a document reaches any API.
The confidence gates in production and retrieval with decisions pieces supply the rest of the design: confidence thresholds per clause type, a human review path for low-confidence extractions, and a retrieval layer that cites the source passage rather than asking the model to remember it. A legal pipeline earns trust by making each of those steps inspectable, not by claiming the model reads like a lawyer.
