Lesson objective

Explain the four things that change when one careful person doing an AI-assisted task once becomes many people doing it every day, and identify which of those changes create new risk rather than more of the old risk.

Plain-language explanation

Why this matters now

Level 2 taught you to do one thing well. Draft a clause, summarise a deposition, check a citation, build a fact packet, sign the work off. Every method in that level assumed one competent person, one matter, one deliberate decision to use a tool, and one reviewer who could still see the whole of the work.

Almost nothing in a firm stays at that scale. A method that works becomes a method that spreads. The associate who found a good way to produce first-pass issue lists tells two colleagues. The practice group adopts it. Six months later thirty people run it on four hundred matters a month, half of them have never read the original instructions, and the partner who signed off the first ten outputs now signs off a summary of a summary.

The scale is not hypothetical. Industry survey: the International Legal Technology Association's 2025 Technology Survey, published 16 September 2025, reported 80% of responding firms using or exploring generative AI, rising to 100% of firms with 700 or more lawyers, while 19% had no policy and a further 25% had one still in development. Those responses came from 580 firms representing about 152,000 attorneys, and the respondents were firm technologists rather than practising lawyers, so read the figures as a picture of what IT departments see rather than a census of practice. The gap that matters for this module is the one between how widely the tools have spread and how far the governing documents have caught up.

Professional guidance: ABA Formal Opinion 512 (29 July 2024) treats a firm's policies and practices on generative AI as part of how supervisory obligations are met, alongside competence, client communication, candor and billing. Law / rule: Model Rules 5.1 and 5.3 place responsibility on supervising lawyers for work done by other lawyers and by non-lawyer assistance. These are model provisions; your jurisdiction adopts its own numbering and wording and you must check what actually binds you.

A question to sit with

The verification method you wrote in Level 2 worked because you did it yourself, on your own matter, while you still remembered why each step was there. What, precisely, is different when thirty people run it on four hundred matters and nobody remembers?

What goes wrong

Frequency turns a small error rate into a certainty. A method that is wrong one time in fifty is a rare and recoverable event when it runs twice a month. The same method run four hundred times a month produces roughly eight bad outputs a month, every month, forever. Nothing about the method changed. The only thing that changed is how often you rolled the dice.

Variance becomes the product. One person running a method varies a little. Thirty people running a method that lives in someone's memory vary enormously, and they vary in ways nobody can see, because each person believes they are doing the standard thing. The firm no longer has a method. It has thirty methods wearing the same name.

The verification step quietly evaporates. This is the most reliable failure in the whole of legal AI operations. The checking step feels wasteful precisely in proportion to how often it finds nothing. A reviewer who has checked two hundred outputs and found two problems learns, correctly on the evidence available to them, that checking rarely pays. They check faster. Then they sample. Then they skim. The step is still on the page and it is no longer happening, and nothing in the record shows the difference.

Distance grows between the work and the reviewer. At one matter the reviewer sees the source, the instruction and the output. At scale the reviewer sees an output, then a batch of outputs, then a report about a batch. Each step is a reasonable response to volume, and each step removes information that supervision depended on.

Failures stop being independent. Thirty lawyers making their own mistakes make thirty different mistakes, and colleagues catch each other. Thirty lawyers running one shared prompt against one shared document set make the same mistake at the same time, in every matter at once, and nobody catches anything because everything agrees.

The practical answer

Start by separating the risks that scale multiplies from the risks that scale creates. They need different responses.

Risk Present at one matter? What scale does Response
Model invents an authority Yes Multiplies it Verification, unchanged from Level 2
Confidential material sent to a tool Yes Multiplies it Access control and tool approval
Method drifts between people Barely Creates it Written method, one owner
Verification silently stops Barely Creates it Measurement and sampling
Every matter shares one defect No Creates it Change control, staged rollout
Nobody owns the outcome No Creates it Named ownership

The right-hand column is the syllabus of this level. Rows three to six are what "operations" means, and none of them can be fixed by a better prompt.

Ask four questions of any task you are about to scale. This is the diagnostic that the rest of the level builds on.

Question Why it matters If the answer is bad
How often will this run? Frequency converts a rate into a count Compute the expected number of bad outputs per month and say it out loud
Who will run it? Variance grows with people and falls with documentation Write it down before you widen it (module 4)
How far is the reviewer from the source? Supervision needs information, not summaries Keep a sampled path back to source material
What do all the runs share? Shared inputs create correlated failure Version and change-control anything shared

Do the frequency arithmetic before the pilot, not after. A useful habit: take the error rate you believe you have, multiply by the monthly volume you intend, and write the answer as a sentence. "At two percent and four hundred matters a month, we expect about eight defective outputs a month." If that sentence is unacceptable, the question is not whether to use the tool but which control changes the number — a narrower task, a heavier review gate, a smaller rollout, or a different step in the workflow altogether. AI Coalition Network recommendation: write that sentence into the pilot proposal itself, so that the decision to proceed is made with the number visible rather than discovered later.

Treat anything shared as a change-controlled asset. A prompt used by thirty people is not a note. A precedent corpus queried by a whole practice group is not a folder. Best practice: shared prompts, shared corpora, shared templates and shared tool configurations get a version, an owner, a change record and a way to roll back. Modules 5 to 7 build this for retrieval; the principle is general.

Widen in stages and watch between stages. Emerging practice: firms increasingly move a method from one team to several in deliberate steps, with a period of heavier sampling at each step, rather than announcing availability to everybody at once. The point of the staging is not caution for its own sake. It is that a defect found at ten users costs ten reworks and a defect found at three hundred costs three hundred, plus the cost of finding out which of the three hundred were affected.

Know what this module is not doing. It is setting up the level, not delivering it. Mapping the workflow you are about to scale is module 2. Deciding how much control a given task deserves is module 3. Writing the method down so a supervising attorney can enforce it is module 4. Measuring any of it without corrupting the measure is module 14. What this lesson asks of you is only that you stop treating "it worked when I did it" as evidence that it will work when the firm does it.

Remember this

  • Scale multiplies some risks and creates others; the created ones — drift, silent loss of verification, correlated failure, unowned outcomes — are what operations exists to handle.
  • Frequency turns an error rate into a count. Do the arithmetic before the rollout and write the expected number of bad outputs in a sentence.
  • The verification step decays fastest when it works, because finding nothing teaches people it is not worth doing.
  • Anything shared by many people — a prompt, a corpus, a template, a configuration — is a change-controlled asset with a version and an owner.
  • Supervision duties do not scale by being restated. If keeping the standard requires a name on work nobody examined, change the workflow.

Legal example

Trenholm Vasquez LLP is a fictional 90-lawyer firm with an invented commercial litigation group. Senior associate Priya Ramanathan builds a method for producing first-pass chronologies from produced documents. She runs it herself on four matters, checks every entry against the underlying document, finds three errors in the first matter and none in the fourth, and refines the instruction each time. By any reasonable measure the method is sound: she can show her sources, her checks, and a named partner sign-off on each chronology. Her practice group leader, the fictional Desmond Okonjo, asks her to roll it out to the group. Nine months later, twenty-two people are using it across roughly 260 matters a year. Two things have happened that nobody decided. First, the instruction has forked. Priya revised her prompt twice; the version most of the group uses is the one she emailed in month two, and four people use a version a paralegal improved and shared informally. Nobody can say which matters used which. Second, the verification step has thinned. Priya checked every entry. The group's written guidance says to check every entry. In practice, most users check the first page and spot-check the rest, because in nine months of checking almost nothing has been wrong. When a defect finally surfaces — the method mis-attributes the author of any document whose header spans two lines, a pattern common in one client's email exports — it is not in one chronology. It is in every chronology produced for that client since the method was adopted, because every matter used the same instruction against the same kind of export. Desmond asks the two questions that matter and gets no answer to either: how many matters are affected, and which version of the instruction did each of them use. The failure was not in Priya's method. It was in the firm's assumption that a method that is safe in one pair of hands is safe in twenty-two.

Responsible-use warning

Never treat a successful pilot as evidence that a method is safe to scale. A pilot run by its own author, on matters they chose, with verification they personally performed, tests the method under the one condition that will not exist in production. The risks that scale creates — drift between users, silent decay of the checking step, a shared defect reaching every matter at once, and an outcome nobody owns — are invisible in a pilot by construction, because a single careful operator suppresses all four. Ask what breaks at thirty users before you ask whether it worked at one.

Human-review requirement

A licensed attorney owns every output that reaches a client or a court, and scale does not divide that ownership among the people who ran the workflow. At the operations level the reviewing attorney owns two further judgements: whether the review actually performed still matches the review the method describes, and whether the volume being produced is within what a named reviewer can genuinely examine. Where it is not, the attorney's obligation is to narrow the scope, add reviewers, or stop the rollout — not to sign a record that overstates what was checked. A sign-off that cannot be honestly given is a stop condition, not a formality.

Practical exercise

Analyse one of your own working methods for what would break at scale. Use only fictional or non-confidential material.

  1. Pick a method you personally use with an AI tool and can describe in a page. Write that page, including every step you actually perform, not the idealised version.
  2. Estimate your current volume for the method, in runs per month, and the volume the firm would reach if the whole practice group adopted it.
  3. Estimate your error rate honestly: of the last twenty runs, how many needed a correction you caught? Multiply the rate by the projected volume and write the result as one sentence.
  4. Identify every input shared across runs: a prompt, a template, a checklist, a document set, a tool configuration. For each, write who would own it and how a change to it would be recorded.
  5. List the steps in your method that exist only to catch errors. Beside each, write what would tell you that people had stopped doing it, using evidence that does not depend on asking them.
  6. Identify the single shared input whose defect would reach the largest number of matters at once. Write two sentences on how you would find out, and how quickly.
  7. Write the three questions you would want answered before agreeing to roll this method out to twenty colleagues, and note which module of this level answers each.

Knowledge check

  1. Which risks does scale multiply, and which does it create?

    Answer guidance: Scale multiplies risks that already exist at one matter, such as invented authorities or confidentiality exposure. It creates risks that barely exist at one matter: drift between users, silent decay of the verification step, correlated failure from a shared input, and an outcome nobody owns. The created risks need operational controls, not better prompts.

  2. Why does the verification step decay fastest when the workflow is working well?

    Answer guidance: Because checking that finds nothing teaches the checker, on the evidence in front of them, that checking rarely pays. They check faster, then sample, then skim. The step remains written down while it stops happening, and nothing in the record distinguishes the two states.

  3. Thirty lawyers run one shared prompt against one shared document set. Why is that riskier than thirty lawyers each working their own way?

    Answer guidance: Because their failures stop being independent. Different people noticing different things is a firm's ordinary defence; a shared instruction removes it. One defect in the shared input reaches every matter at the same time, and consistency across outputs looks like quality rather than like a common cause.

  4. What does the frequency arithmetic add to a rollout decision?

    Answer guidance: It converts an abstract error rate into a number of defective outputs per month at the intended volume, stated in a sentence before the decision is made. That number is what makes the trade-off discussable: if it is unacceptable, the question becomes which control changes it, rather than whether the tool is good.

Sources and further reading