Lesson objective
Diagnose the common reasons a legal AI pilot fails to become routine practice, and design a pilot whose result is a decision rather than an impression.
Plain-language explanation
Why this matters now
The gap between trying a technology and running it is where most firm AI programmes currently sit. Industry survey: the ILTA 2025 Technology Survey (16 September 2025) reported 80% of responding firms using or exploring generative AI and 100% of firms with 700 or more lawyers, alongside 19% with no policy and 25% with one still in development. The phrase "using or exploring" is doing real work in that sentence, and the distinction between the two is the subject of this lesson. The survey drew on 580 firms and about 152,000 attorneys, and its respondents were firm technologists rather than practising lawyers.
Stalling is not usually a technology failure. The pilot generally works. What fails is the transition: the pilot ends, the enthusiasm disperses, no one owns the result, no method is written, no risk decision is recorded, and six months later the firm runs another pilot on the same task with different people.
That pattern is expensive in a way that is easy to miss. Each cycle consumes senior attention, which is the scarcest resource a firm has for this work, and each cycle teaches the practice group that these initiatives do not lead anywhere. The third pilot is harder to staff than the first.
A question to sit with
Your pilot succeeded. Write the sentence that says what the firm decided as a result. If you cannot write it, what exactly did the pilot produce?
What goes wrong
The pilot was designed to impress, not to decide. Volunteers who wanted it to work, matters chosen for suitability, no baseline measurement of the old way, and an evaluation that asks participants whether they liked it. Such a pilot cannot fail, which is another way of saying it cannot inform.
No baseline. Without knowing what the task cost, how long it took and how often it went wrong before, "it saved time" is an impression. There is nothing to compare, and the claim cannot survive the first sceptical partner.
Benchmark scores are mistaken for readiness. Industry benchmark: the Vals Legal AI Report (Vals AI, 27 February 2025) is a task-based benchmark run against a lawyer baseline across four tools, in which vendors could opt in or out — one major vendor withdrew from all skills except research. Its scores on zero-shot document tasks are not hallucination rates. A benchmark tells you something about a tool under test conditions chosen by the benchmark. It tells you nothing about your documents, your matters, your people or your review gate, and a vendor-participating benchmark carries selection effects that must be stated whenever a score is quoted.
Nobody owns it on the Monday after. The pilot had a sponsor, a working group and an end date. Production needs an owner, and the pilot rarely appoints one, so the capability has no address at the moment it needs one most.
No route to production exists. Even a firm that decides yes often has no next step defined: no SOP, no risk tier, no training, no access provisioning, no support route, no measurement, no fallback when the tool is unavailable. The decision is made and nothing moves, and the absence of movement is read as a change of mind.
The commercial conversation was never had. Industry survey: the Thomson Reuters Institute report "The great AI disconnect" (18 March 2026) found 68% of corporate legal professionals did not know whether their outside firms use AI, and that most departments had not discussed AI use or its billing effect with their firms. A pilot that produces time savings but no position on what clients are told and how the bill changes stalls at exactly the point where a partner has to answer a client question and would rather not.
The policy does not cover it. A firm with no policy, or one still in development, cannot approve routine use without either writing the policy or making an exception nobody wants to sign.
The practical answer
Design the pilot backwards from the decision. Write the decision first, then design the smallest test that could produce it.
| Element | Weak pilot | Pilot that decides |
|---|---|---|
| Question | Does this tool work? | Should we make this task an operation for this group, at this volume? |
| Participants | Volunteers who wanted it | A mix including at least one sceptic and one average performer |
| Matters | Chosen for suitability | Chosen to include the awkward ones |
| Baseline | None | Time, cost and defect rate for the current method, measured first |
| Outcome measure | Satisfaction | Defect rate against a defined taxonomy, plus time, plus review burden |
| Result | A report | A written decision with a named owner and a date |
Measure the review burden, not just the drafting time. A workflow that halves drafting and doubles the checking has not saved anything; it has moved work from a junior person to a senior one, which is usually more expensive. AI Coalition Network recommendation: every pilot records reviewer minutes per deliverable alongside producer minutes, because the review time is what determines whether the workflow is sustainable at volume.
Use a production readiness gate. A pilot may only become practice when every line has an answer. Best practice:
| Gate item | Answered by |
|---|---|
| Named method owner and named deputy | This module |
| Workflow mapped, inputs and outputs defined | Module 2 |
| Risk tier assigned by consequence and reversibility | Module 3 |
| Written SOP a supervising attorney can enforce | Module 4 |
| Review gate defined, with reviewer capacity checked against volume | Modules 3 and 4 |
| Measurement defined, including one measure that can get worse | Module 13 |
| Escalation trigger, address, time limit and matter state | Module 14 |
| Client communication and billing position stated | This module and firm policy |
| Policy coverage confirmed, or an exception recorded | Firm policy |
| Fallback when the tool is unavailable or withdrawn | This module |
Keep the fallback honest. Any operation that depends on a vendor needs an answer to the tool disappearing, changing behaviour, or being withdrawn mid-matter. Best practice: the fallback is written before the dependency is created, because it is written calmly then and not at all later.
Give the programme a shared vocabulary. Best practice: the NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, 26 January 2023) offers Govern, Map, Measure and Manage as four words a mixed group of lawyers, technologists and risk staff can share. It is voluntary, and adopting it neither creates nor discharges a professional duty; its value is that it stops four functions being discussed as one.
Sequence the level from here. Module 2 maps the workflow you are about to make an operation. Module 3 tiers it by consequence and reversibility. Module 4 writes the SOP. Modules 5 to 7 build and govern the knowledge the workflow draws on; modules 8 to 12 apply all of it to discovery, privilege, trial preparation, diligence and regulatory monitoring; modules 13 and 14 measure it and give it stop conditions. Nothing in this module is finished. It is the argument for why the rest is worth the effort.
Remember this
- Stalling is a transition failure, not a technology failure; the pilot usually works and the handover does not.
- Design the pilot backwards from the decision you need, include a sceptic and the awkward matters, and measure a baseline before you start.
- A benchmark score, including a vendor-participating one, says nothing about your documents, your people or your review gate.
- Measure reviewer minutes as well as producer minutes; a workflow that moves work to a more expensive person has not saved anything.
- Nothing becomes practice until the readiness gate is answered: owner, map, tier, SOP, review gate, measurement, escalation, client position, policy coverage and fallback.
Legal example
Kestrel Anand LLP is a fictional 150-lawyer firm whose invented innovation committee has run three pilots of AI-assisted contract abstraction in two years. None reached production, and the committee chair, the fictional Rosalind Achebe, is asked to explain why before a fourth is approved. She reads the three reports together. Pilot one, eighteen months earlier, used six volunteers from the corporate group on matters they selected. It reported strong satisfaction, an estimated 60% time saving and a recommendation to proceed. There was no baseline, so the 60% was an estimate by the participants of how long the work used to take. The report named no owner and no next step. Pilot two changed vendors on the strength of a published benchmark score, ran for eight weeks and ended when its sponsor moved practice groups; nothing was written down about what it found. Pilot three was better designed, included two sceptics, and measured defect rates against a short taxonomy. It found the workflow sound but reported that reviewing attorney time per abstract rose from twelve minutes to twenty-nine, because the abstracts were longer and the reviewer could not tell which entries had been verified. That is a genuine and useful finding, and it stalled the programme, because nobody owned the question of how to fix it. Rosalind writes a one-page recommendation. No fourth pilot. Instead, take pilot three's workflow, fix the single defect it identified by marking verified entries, and put it through the readiness gate: name an owner and a deputy, map the workflow, assign a risk tier, write the SOP, define the review gate against realistic volume, define one measure that can get worse, write the escalation rule, state the client communication and billing position, confirm policy coverage and write the fallback. Her closing line is the one the committee remembers: the firm does not need more evidence that the tool works, it needs one task it actually runs.
Responsible-use warning
Never quote a benchmark score as evidence that a workflow is ready for your firm. A benchmark measures a tool on tasks the benchmark chose, under conditions it set, and where vendors participate voluntarily some tools will be absent or partially represented — a selection effect that must be stated whenever a score is used. It says nothing about your document population, your matter types, your people, or the review gate that will actually catch errors. Benchmarks help you shortlist tools and ask vendors better questions. The evidence that a workflow is ready comes from your own pilot, with your own baseline and your own defect taxonomy.
Human-review requirement
A licensed attorney owns every deliverable produced during a pilot, exactly as in production; pilot status lowers no professional duty. At the transition point the supervising attorney owns a further judgement: whether the review gate the pilot relied on can be sustained at production volume by the people who will actually be available. A pilot reviewed personally by the partner who championed it proves nothing about a workflow reviewed by whoever is free that week. Where the review burden the pilot measured cannot be met at scale, the honest response is to narrow the scope or add reviewers, not to proceed and let the gate thin out in practice.
Practical exercise
Redesign a pilot so that it produces a decision. Use only non-confidential material and a fictional or anonymised task.
- Name a task your firm has piloted, or would pilot, with AI assistance. Write the decision sentence you want at the end, in the form "We will or will not make X an operation for group Y, at volume Z, with review gate R."
- Define the baseline you would need: how long the current method takes, what it costs, and how often it produces a defect. State how you would measure each before the pilot starts.
- Write a defect taxonomy of five to eight categories specific to this task. Keep each category observable by someone reviewing an output.
- Choose participants: name the roles, and include at least one sceptic and one average performer. Write one sentence on why volunteers alone would bias the result.
- Choose matters, including two that are awkward for the workflow. Say what makes them awkward.
- Define two time measures: producer minutes per deliverable and reviewer minutes per deliverable. Say how each is captured without relying on memory.
- Write the readiness gate for this task as a checklist, using the ten items from the lesson, and mark which you could answer today.
- Write the fallback: what the group does for this task if the tool is unavailable for a week, and who decides.
- In three sentences, state what result would make you recommend against production, and be specific enough that the recommendation could actually happen.
Knowledge check
-
Why is a pilot staffed entirely by volunteers a weak basis for a rollout decision?
Answer guidance: Because volunteers wanted it to work, and their result reflects motivated users on matters they chose. Production will be run by average performers on the matters that arrive, including awkward ones. A pilot needs at least one sceptic and some difficult matters, or a negative result is impossible and the exercise cannot inform a decision.
-
What does a task-based benchmark score tell you about your firm, and what does it not?
Answer guidance: It tells you something about a tool on tasks the benchmark chose under conditions it set, among the vendors that chose to participate. It tells you nothing about your document population, your matter types, your people or your review gate, and where the benchmark is vendor-participating the selection effect must be stated whenever a score is quoted.
-
Why must a pilot measure reviewer minutes as well as producer minutes?
Answer guidance: Because a workflow can halve the drafting time while doubling the checking time, which moves work from a junior person to a senior one and usually costs more. Review time is also what determines whether the workflow is sustainable at production volume, so a pilot that measures only production time can recommend something that cannot actually be run.
-
A firm has five successful pilots and nothing in production. What does it actually have?
Answer guidance: Five reports and no organisational capability. A firm with one task running as a genuine operation — owned, mapped, tiered, written, reviewed, measured and escalatable — can apply that machinery to its second task quickly. The firm with five pilots starts from the beginning each time, and each cycle makes the next one harder to staff.
Sources and further reading
- International Legal Technology Association — ILTA 2025 Technology Survey (2025-09-16). Accessed 2026-09-14.
- Vals AI — Vals Legal AI Report (VLAIR) (2025-02-27). Accessed 2026-09-14.
- Thomson Reuters Institute — The great AI disconnect: Law firms & legal departments are not communicating about AI usage (2026-03-18). Accessed 2026-09-14.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023-01-26). Accessed 2026-09-14.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-14.
- American Bar Association Standing Committee on Ethics and Professional Responsibility — Formal Opinion 512: Generative Artificial Intelligence Tools (2024-07-29). Accessed 2026-09-14.