The problem: which model, for which task?
One of the recurring difficulties in legal AI is deciding which model to use for a given task. The set of available models is large, and their strengths are uneven. One model drafts well, another is stronger at closed-universe analysis, and a third leads on a particular practice area. These rankings shift with almost every release.
Public benchmarks do not resolve this for a firm. They measure a general notion of quality on tasks that are not the firm's own. The question a firm needs to answer is narrower and more consequential: on the work it actually does, judged by its own standards, which model should it use?
The only reliable way to answer that question is to construct the benchmark internally. This is why we built a benchmark suite into LegalWork. It lets a firm define its own standard, create its own tasks, specify what counts as good and what does not, and evaluate any model against them. The result is a defensible basis for choosing a model per task and per practice area.
What a benchmark task is made of
Within LegalWork, the benchmark is found under Learning, then Benchmark, and is organized into three tabs: Tasks, Benchmark runs, and Your Leaderboard. A benchmark is a set of tasks, and each task has four components.
- Instruction. A description of what the model is expected to do. It is the same prompt a lawyer would provide when working with the model directly.
- Input documents. The files the model must search in order to complete the task, meaning the full set of material it needs to do the work.
- Deliverable. The output file the model is expected to produce, for example a
memo.docx. - Pass and fail criteria. The most important component, and the firm's explicit definition of quality. Each criterion is written in the form PASS if …, FAIL if …, and a single task can carry many of them. One draft-emergency-application task in our demo defines 62 pass criteria, capturing both what the model should do and what it should avoid.
Running a task is not a separate mode of operation. It is an ordinary agent session, the same one a lawyer already uses in LegalWork, with the difference that its output is graded against the criteria.
Creating and importing tasks
To define a standard from scratch, a user creates a new task and supplies each of the components above: the title, the task type, the instructions, the input documents, the deliverable filename, and the pass and fail criteria that encode the firm's view of quality.
Firms that prefer not to author every task can import from predefined benchmarks. LegalWork includes the Harvey Legal Agent benchmark, which comprises roughly 1,700 tasks. These can be browsed and filtered so that a firm selects the tasks closest to its own day-to-day work.
Running a benchmark
Once tasks have been created or imported, the user selects the relevant ones, names the run, and chooses the models to evaluate. Any model connected through the firm's configured AI providers is available. In the demo, we selected four: DeepSeek V4 Flash, DeepSeek V4 Pro, Gemini 2.5 Flash, and Gemini 2.5 Pro.
Realistic benchmarks take time, because each task runs as a full agent session and typically requires 20 to 30 minutes. That is the cost of measuring performance on genuine work rather than on simplified prompts. The results below come from a completed demo benchmark of three tasks, comprising 168 criteria in total.
In this run, DeepSeek V4 Pro finished first, followed by DeepSeek V4 Flash and then the two Gemini models. This ordering is not a general statement about the models. It holds for these three tasks, graded against these criteria. A different set of tasks or criteria can produce a different order, which is precisely the reason to run the evaluation on the firm's own work.
Reading the results
The top of a benchmark run presents the overall model ranking across every task in that run. Below it, the results are broken down by task and by model, and any individual criterion can be opened to see how a given model performed against it.
The view also supports full inspection. Opening a single task for a single model reveals everything the model did: the complete agent session, the deliverable it produced, and every grading decision, including the reason a criterion was recorded as passed or failed. No part of the score is opaque.
At the foot of the run, results are aggregated by practice area. Because tasks can be tagged, models can be compared within each area. This matters, because a model that performs well in one practice area often performs less well in another. At this level, model selection becomes a routing decision rather than a single leaderboard position.
The leaderboard
The Your Leaderboard tab aggregates every run into one view, across all tasks and all benchmarks. It can be filtered by practice area to show which model performs best in the area a firm actually works in. As more tasks are added and more models are evaluated, this view becomes a continuously updated record of which model to rely on, and where.
Own your evaluation
A firm that intends to own its models should also own its evaluation. The pass and fail criteria are where a firm records its judgment, including its house style, its risk posture, and its standard for what a partner would be willing to send to a client, in a form against which a model can be measured. Once that standard exists, model selection no longer depends on vendor claims or general leaderboards. It becomes a decision the firm can justify on the basis of its own work.
The benchmark suite is fully open source and free within the LegalWork application. Firms that want to define their own standard can download LegalWork and begin with a task whose correct answer is already known.
Running an evaluation at the scale of a full firm, across practice areas, matter types, and private data, is a larger undertaking. Eigenwelt Labs works with firms to design and run these evaluations on their own matters, with the data and results staying inside the firm. If that is the problem you are facing, we would like to hear from you.