What we are testing
Jev is TypeSafe AI's decision model. Given a document and a set of questions, it returns probabilities over predefined answers in a single pass. Low latency and cost make this approach useful for repeated decisions in tabular review, document search and citation checks.
The open question is answer quality. We compare Jev with two open alternatives, Laya and DiffusionGemma, and a fast autoregressive LLM, Gemini 3.8 Flash. Two lawyer-annotated benchmarks test different parts of contract review: MAUD tests interpretation of a supplied clause; our adaptation of CUAD tests retrieval of relevant paragraphs.
The main result: Jev outperformed the open alternatives without task-specific fine-tuning. Fine-tuned Laya achieved higher mean AUPR than Jev at lower reported cost and latency. Gemini achieved the highest mean AUPR, with cost strongly dependent on request design.
How decision models differ from LLMs
A decision model returns a probability for each allowed option. An autoregressive LLM generates text one token at a time. That text can follow a schema, but a confidence value the model writes is a generated estimate, rather than a direct probability over the supplied options. Fixed outputs simplify integration; they do not guarantee correct answers.
The cost difference can be substantial. On MAUD, Jev cost $0.0315 per 1,000 decisions, compared with $2.39 for Gemini: approximately 1/76 of the cost. Median request latency was 293 ms versus 3,065 ms. These ratios apply to the tested configuration. On CUAD, processing a whole contract in one Gemini request changed the comparison considerably.
Models
Jev. A hosted decision model, billed by input text. Its architecture and parameter count were not available for this comparison.
Laya. A 421-million-parameter model from Convai Innovations, built on ModernBERT with a decision head. Its base performance was weak, so we fine-tuned it separately on each benchmark's training data. Each run took under 40 minutes on a single graphics card. We report both base and fine-tuned results. The fine-tuned runs used a 4,096-token input limit.
DiffusionGemma. Google's 26B-parameter model, served through OpenJev. The server converts scores for allowed answers into probabilities without generating a text response. Like Laya, it can run on the operator's own hardware.
Gemini 3.8 Flash. Google's general-purpose LLM, prompted to return answers in a fixed schema. It generates one token at a time. CUAD includes both paragraph-by-paragraph and whole-contract configurations.
MAUD: interpreting a clause
Interpreting the clause.
144 scoring targets · 5,223 requests
MAUD contains 152 merger agreements and 92 questions annotated by lawyers. Each question supplies the relevant passage and a fixed set of answers, covering issues such as material adverse effects and closing conditions. Our evaluation used 144 scoring targets and 5,223 test requests.
The primary metric is mean area under the precision-recall curve (AUPR), averaged across scoring targets. Higher is better. This measures how well the model ranks correct answers across decision thresholds. Balanced accuracy and calibration describe different aspects of performance; they are available in the chart above.
Gemini led with an AUPR of 0.699, followed by fine-tuned Laya at 0.684 and Jev at 0.656. Fine-tuning raised Laya from 0.302 to 0.684. MAUD's official split shares contracts across training and test, and 91.5% of test decisions use an excerpt also present in training. The result therefore does not establish performance on unseen contracts.
Fine-tuned Laya also had the lowest calibration error, at 0.096. Jev's error was 0.205; DiffusionGemma's was higher at 0.250. Lower calibration error means closer agreement between predicted confidence and observed outcomes.
CUAD: finding a clause
Finding the clause.
40 scorable clause types · 500,282 decisions
CUAD contains 510 commercial contracts, 41 clause types and 13,823 lawyer-annotated passages. The official task asks for the exact text of a clause. We adapt it to paragraph classification: split contracts at blank lines, then score each paragraph against all 41 clause types. A paragraph is positive if it overlaps an annotated span of that type.
The official test split yields 12,202 paragraphs and 500,282 decisions. Only 0.41% are positive. Mean AUPR is calculated over 40 clause types; Price Restrictions has no positive test examples. These scores are not directly comparable with published CUAD span-extraction results.
Paragraph-based Gemini led at 0.644. Fine-tuned Laya scored 0.620, compared with 0.553 for Jev. Laya exceeded Jev on 27 clause types and trailed it on 13. Base Laya scored only 0.050.
DiffusionGemma's weakness was most pronounced among its highest-ranked results. It ranked a positive paragraph above a negative one about 93% of the time, versus 95% for Jev. But only 47% of its five highest-scoring paragraphs per clause type were relevant, compared with 78% for Jev. Broad ranking performance masked weaker precision among the results a reviewer would see first.
Calibration needs care in this setting. DiffusionGemma had lower calibration error than Jev despite substantially lower AUPR. When almost all decisions are negative, low overall calibration error can coexist with poor retrieval.
Quality, cost and latency
Quality, at what cost?
Explore the measured frontier for cost and request latency.
The Pareto frontier identifies configurations for which no other tested option offers at least as much AUPR at no greater cost or latency, with an improvement in one of those dimensions. The chart separates the two benchmarks and lets you include or exclude task-specific fine-tuning.
Fine-tuned Laya offers a strong quality-cost trade-off on both tasks. Gemini reaches higher AUPR at higher reported cost. Excluding fine-tuning changes the frontier, which matters when labeled training examples are unavailable.
Request design changes the economics. On CUAD, whole-contract Gemini scored 0.615 at $0.0044 per 1,000 decisions, close to Jev's $0.0038. Paragraph-based Gemini cost $0.2052. The whole-contract run has more context and amortizes work across decisions, so it is a separate configuration.
A whole-contract request also covers more work than a paragraph request. We show it separately in the latency plot and exclude it from the paragraph-request frontier. Request latency alone does not establish throughput.
What the results suggest
For teams with labeled examples, a small model fine-tuned on the task is worth evaluating: Laya exceeded Jev's AUPR on both benchmarks at lower reported inference cost and latency. Without task-specific training, Jev performed better than the tested open alternatives. Gemini led on AUPR, and its whole-contract CUAD result shows why request design should be part of any cost comparison.
Scope and limitations
These benchmarks cover English contracts filed with the SEC. They do not establish performance in other languages or jurisdictions, on other document types, or on noisy document extraction. MAUD supplies the relevant passage; our CUAD adaptation evaluates paragraph detection rather than exact span extraction. Neither benchmark measures the quality of explanations or supporting citations.
Error bars show 95% bootstrap confidence intervals across targets, using 1,000 replicates. They describe variation across questions, not generalization to new contracts. CUAD separates contracts across splits; MAUD does not.
Self-hosted costs use GPU busy time on one RTX PRO 6000 Blackwell (96 GB) at an assumed $1.80/hour, excluding training and idle time. API costs use metered charges for Jev and token-based list prices for Gemini. Latency is the median per request: Laya is measured locally, while the others include HTTP. Concurrency differs, and overlapping launches affected some CUAD timings, so these are configuration comparisons rather than a controlled speed ranking.
Sources
Let’s discuss the results.
Join our webinar on Jev, small models and the trade-offs behind fast legal AI. We’ll walk through the results and take your questions.
Register for the webinar