Lesson objective

Interrogate a benchmark score or a vendor performance claim by asking what was measured, against what baseline, on whose material, and who chose to participate.

Plain-language explanation

Why this matters now

Level 2 asks you to decide which tool to use for a task. That decision is made in a market full of numbers, and most of those numbers are supplied by the people selling the tool.

Benchmarks exist. The Vals Legal AI Report, published February 27, 2025 by Vals AI, is a task-based benchmark of legal AI tools measured against a lawyer baseline. It is useful precisely because it makes its own structure visible: it covered four tools, vendors could opt in or out, and one major vendor withdrew from every skill except research. Understanding what that structure means is the difference between reading a benchmark and being sold by one.

The context makes it urgent. The Thomson Reuters Institute's 2026 report, published February 9, 2026, put organisational generative AI use at 40% while only 18% of respondents said they knew their organisation tracked return on investment, and put agentic AI use at 15%. Clio's 2025 Legal Trends Report, published October 16, 2025, described adoption plateauing at 79%, with 45% of wide adopters having adjusted prices and 36% reporting a positive revenue effect. The profession is buying and pricing around claims. Very little of it is measuring.

A question to sit with

A vendor tells your firm its tool "outperforms lawyers on contract review." A partner asks you to say, in two minutes, whether that sentence means anything. What are the first three questions out of your mouth?

If your first question is "by how much," you have already accepted the frame. The prior questions are what was measured, against whom, and who decided to be measured.

What goes wrong

  • Score transfer. A score earned on one task type is quoted as general capability. A tool good at extracting dates from a contract is not thereby good at forming a view on a novel question.
  • Missing baseline. "90% accurate" against what? A benchmark with a stated human baseline is telling you something; a percentage with no comparator is telling you nothing.
  • Participation bias. Where vendors opt in, the field you are reading is the field that chose to be read. The Vals report is explicit that vendors could opt in or out and that one major vendor withdrew from all skills except research — a fact that is only visible because the report says so.
  • Accuracy read as hallucination rate. These are different measurements. A score on a zero-shot document task is not a fabrication rate, and the Vals report itself cautions against reading it as one.
  • Survey figures treated as your figures. A cross-industry adoption number describes a population, not your matter, your data or your risk.
  • Respondent mismatch. ILTA's 2025 survey drew on 580 firms representing about 152,000 attorneys, but its respondents are firm technologists rather than practising lawyers. That is the right lens for infrastructure and policy questions and the wrong one for "what does this feel like in a deposition."
  • Vendor survey read as independent research. Clio's Legal Trends Report is published by a legal software vendor. That does not make it wrong; it makes the interest disclosable and the methodology worth reading.

The practical answer

The six questions to ask of any performance claim

# Question What a weak answer sounds like
1 What exactly was measured — which task, on what material? "Legal work."
2 What was the baseline, and who set it? "Industry-leading."
3 Who chose to participate, and could anyone decline? Silence, or "leading vendors."
4 Is this an accuracy score or an error-rate measure? They are not the same. The two used interchangeably.
5 Who published it, and what do they sell? "Independent" with no publisher named.
6 Does the tested configuration match the one I would actually use? "The enterprise version is similar."

Matching the evidence to the question

If you want to know… A task benchmark tells you An adoption survey tells you Your own pilot tells you
Can this tool do my task? Something, if the task matches Nothing Almost everything
Is my firm unusual? Nothing A lot Nothing
How often will it invent something? Only if it measured that Nothing A lot, with enough samples
Should we change our pricing? Nothing Context only The only honest basis

Run your own small evaluation. The most useful number about a tool is the one you produce. It does not need to be elaborate:

  1. Pick one task you actually do, in one mode from Lesson 1.
  2. Build ten fictional or de-identified samples with known correct answers.
  3. Run all ten through the tool with the same instruction.
  4. Score each output against the known answer on three axes: correct, incomplete, invented.
  5. Read the inventions closely. Their shape — plausible citation, confident date, filled-in silence — tells you what to check forever after.
  6. Write four sentences: what it does well, what it gets wrong, what you will always check, and what you will never ask it.

Remember this

  • Ask what was measured, against what baseline, on whose material, and who could decline to participate.
  • An accuracy score on a document task is not a fabrication rate.
  • Adoption surveys describe a population. They never describe your matter.
  • Note who published a figure and what they sell; Clio's report is a vendor survey, ILTA's respondents are firm technologists.
  • Ten samples you scored yourself beat any published number for deciding whether a tool fits your task.

Legal example

The fictional firm Halvorsen Pike LLP is considering a fictional contract-review product, ClauseHarbor. The vendor's one-page summary leads with: "Outperforms the human baseline on contract review, with 94% accuracy."

Marcus Oyelaran, the partner sponsoring the evaluation, asks Nadia Blankenship to spend two hours on it before anyone sees a demo. She runs the six questions.

What was measured? A footnote says the figure comes from an internal test of clause identification on commercial agreements. Nadia writes: clause identification, one document type. The firm's actual pain is negotiating position against a playbook, a different task.

What baseline? The footnote compares against "a junior reviewer," without saying how many, how experienced, or how scored. Nadia writes: baseline undefined.

Who participated, and could anyone decline? This is an internal test, so nobody else was in the field at all. She notes the contrast with the published Vals Legal AI Report of February 27, 2025, which measured four tools against a lawyer baseline and states plainly that vendors could opt in or out, and that one major vendor withdrew from all skills except research. The point she takes is structural: a benchmark that tells you who declined is telling you something a vendor's own number never will.

Accuracy or error rate? The vendor uses 94% accuracy and, three lines later, says the tool "eliminates hallucinations." An accuracy score on a clause-identification task measures how often the tool got a clause right; it is not a measurement of how often it invents something. Nadia writes one line: these are two different measurements and only one was made.

Who published it, and what do they sell? The vendor. Disclosed, understandable, and a reason to weight it low. Same configuration? The test used single-document mode; Halvorsen Pike would bulk-upload across a data room, so the number does not transfer.

She then runs a ten-sample evaluation using fictional agreements the firm drafted for training, each with a known set of clauses and three deliberate omissions. ClauseHarbor identifies clauses well. On two of the ten it reports a governing-law clause that is not in the document at all.

Nadia's memo to Marcus ends: "It is good at finding what is there and it will occasionally report something that is not, so if we buy it, every clause it reports gets opened before it is relied on." Marcus takes the demo with that sentence in his pocket.

Responsible-use warning

Never quote a benchmark score, a survey figure or a vendor claim in advice to a client or in an internal recommendation without naming who published it, what exactly was measured, and against what baseline. The two most damaging shortcuts are treating an accuracy score as a hallucination rate — they are different measurements, and only one of them tells you how often a tool invents authority — and treating a population-level adoption figure as if it described your matter, your data or your risk. A number without a methodology is an advertisement wearing a percentage sign.

Human-review requirement

Before a tool is approved for a class of work, a supervising attorney reviews the evaluation rather than the vendor material. The reviewer checks that the six interrogation questions were answered from the published methodology and not from marketing copy, that any accuracy figure is not being read as an error or fabrication rate, that the tested configuration matches what the firm would actually deploy, and that the firm ran its own samples on its own task. The reviewer records the approved uses, the known failure shapes, the checks that catch them, and a re-evaluation date. A licensed attorney owns every deliverable the tool contributes to, whatever any benchmark reports.

Practical exercise

Use only public material and fictional samples you create.

  1. Find one publicly available performance claim about a legal AI tool — a vendor page, a press release, a conference slide. Copy the exact sentence.
  2. Write the six interrogation questions from the lesson down the left of a page.
  3. Answer each one strictly from the source. Where the source does not answer, write "not stated." Do not fill a gap from memory or inference.
  4. Count the "not stated" answers. Write one sentence on how much weight the claim can carry given that count.
  5. Identify whether the claim is an accuracy measure, an error-rate measure, or unclear. If the source uses both words, quote both sentences side by side.
  6. Name the publisher and what they sell. Write one sentence on how that affects the weight, without assuming bad faith.
  7. Design a ten-sample evaluation for one task you actually do: describe the samples, the known correct answers, the single instruction you would use, and the three scoring axes (correct, incomplete, invented).
  8. Build and run at least three of the ten samples using fictional material only. Record what the tool got right, what it missed, and anything it invented.
  9. Write the four sentences from the lesson: what it does well, what it gets wrong, what you will always check, what you will never ask it.
  10. Paste the interrogation sheet and the four sentences into the submission box.

Record in the workbook: the claim, the count of "not stated" answers, and the shape of any invention you saw. Workbook: Module 01 worksheet, Vendor claim interrogation sheet.

Knowledge check

  1. A vendor reports that its tool scored 91% on a document task. Does that tell you how often the tool fabricates authority?

    Answer guidance: No. An accuracy score on a document task measures how often outputs matched the expected answer on that task; it is a different measurement from a fabrication or hallucination rate. The Vals Legal AI Report of February 27, 2025 cautions that scores on zero-shot document tasks are not hallucination rates. If you want a fabrication rate, it has to be measured directly.

  2. Why does it matter that vendors could opt in or out of the Vals benchmark?

    Answer guidance: Because the field you are reading is the field that chose to be read. The report states that vendors could opt in or out and that one major vendor withdrew from all skills except research, so the comparison is between participants, not between everything on the market. A benchmark that discloses this is more useful than one that does not, because you can price the gap.

  3. ILTA's 2025 Technology Survey drew on 580 firms representing about 152,000 attorneys. Why does the identity of the respondents matter when you read it?

    Answer guidance: Because the respondents are firm technologists rather than practising lawyers, and full results are member-only. That makes it strong evidence about infrastructure, deployment and policy status — such as its finding that 19% of responding firms had no policy and 25% had one still in development — and weak evidence about what using a tool feels like inside a matter.

  4. Your firm must decide whether a tool fits one specific task. What single piece of evidence should carry the most weight?

    Answer guidance: Your own small evaluation: ten samples of that exact task with known correct answers, scored as correct, incomplete or invented. It is the only evidence generated on your task, your material and the configuration you would actually use. The AI Coalition Network recommends weighting it above any published score and keeping it with the tool record.

Sources and further reading