Lesson objective

Plan, before a session is delivered, the evidence in the room and afterwards in the work that will show whether the target behaviour occurred.

Plain-language explanation

Why this matters now

The final question in backwards design is the one most sessions never answer: how will you know? Not whether participants enjoyed it, attended it or could recall its content, but whether the behaviour named on the first line of the plan actually happened, in the session and afterwards at the desk.

This lesson sits last in the module, but the evidence is decided at the same time as the behaviour statement. It is the test of whether that statement was observable. If you cannot say what you would see, the behaviour was not specific enough, and you return to the first lesson.

This is Best practice and, for the named-reviewer record, an AI Coalition Network recommendation. Module 10 treats formal assessment in depth. Here the concern is narrower: planning, as part of session design, what evidence the session and the working environment will produce.

There is a professional reason to care as well. Law / rule: ABA Model Rules 5.1 and 5.3 address the responsibilities of lawyers with managerial and supervisory authority for the conduct of other lawyers and of non-lawyer assistants. They are model rules; each jurisdiction adopts its own version with its own numbering and wording. Professional guidance: ABA Formal Opinion 512 (29 July 2024) applies supervision to generative AI and treats firm policies as part of it. A firm that trains people and never learns whether the training changed anything has less to show for its supervision than it thinks.

A question to sit with

Six weeks after your session, a managing partner asks whether it worked. You have a satisfaction score of 4.6 out of 5 and an attendance list. What do those two numbers actually tell the partner about what anyone does at their desk?

What goes wrong

1. Satisfaction is treated as effect. A room that enjoyed a session has told you it enjoyed the session. That is worth knowing and it is not evidence of behaviour.

2. Recall is treated as behaviour. A quiz at the end shows participants can recognise the right answer immediately afterwards. It does not show they recognise the trigger in a real document weeks later.

3. Evidence is an afterthought. Nobody decides in advance what would count, so afterwards there is nothing to look at, or only whatever happens to be lying around.

4. Evidence becomes surveillance. In the search for proof, the firm starts monitoring individual keystrokes or reading people's tool histories without a policy basis. The session loses the trust it depended on.

5. Evidence uses client material carelessly. An instructor reviews real matter files to see whether verification notes exist, without authority to see those files or a reason to see their contents.

6. No change is read as failure of the participants. Evidence shows the behaviour did not happen, and the conclusion drawn is about people rather than about the design, the tools or the policy.

The practical answer

Best practice: plan evidence at three points.

When What you look for Example for a verification behaviour
In the session What participants produced during the practice Did each participant open the authorities and complete the verification note on the fictional brief? How many recorded a check on the accurate items as well as the flawed one?
Soon after Whether the routine appears in real work in the weeks that follow Do verification notes appear on AI-assisted drafts reaching supervisors? Are supervisors seeing them?
Later Whether the failure the behaviour prevents is still occurring Are supervisors still finding unchecked authorities in AI-assisted drafts?

In-session evidence is the easiest and the most under-used. The practice task already produces work. Collect it, anonymously if that protects participation, and look at it before the debrief ends. You will learn whether the design worked in the only setting you control.

After-session evidence should come from records the work already creates. The best evidence is a by-product of the behaviour itself, which is why a behaviour statement with a recorded standard is easier to evaluate than one without.

  • The verification note on a matter file, where the firm's system records whether one exists.
  • The named-reviewer entry on an AI-assisted deliverable.
  • Questions reaching the named contact for tool and policy queries.
  • A supervising lawyer's short, structured report of what they are seeing in drafts.

Best practice: count what the record shows, not what it contains. Evidence about whether a routine is happening rarely needs anyone to read client material. Whether a verification note exists, how many AI-assisted drafts carry a reviewer entry, and whether the numbers move after a session can usually be reported in aggregate by someone already authorised to see the system. Law / rule: ABA Model Rule 1.6 addresses the confidentiality of information relating to the representation; it is a model rule and each jurisdiction adopts its own version. An instructor who is not working on a matter has no reason to see its contents in order to evaluate training, and the evidence plan should be designed so they never need to.

Best practice: agree the evidence plan with whoever owns the data. Before the session, confirm with the firm's general counsel or risk function what records exist, who may see them, in what form, and under which policy. Tell participants what will be looked at and how it will be reported. Evidence gathered openly and in aggregate supports the session; evidence gathered covertly about individuals damages it.

Best practice: decide in advance what result would change the design. Write down, before delivery, what you expect to see and what you would do if you do not see it.

Result Most likely meaning Next step
Practice work shows the routine performed; real records later do not The routine does not survive real conditions: time, tools or policy Investigate the obstacle before re-training anyone
Practice work shows participants missed the trigger The practice announced too much, or too little Redesign the practice
Records show the routine and supervisors still find failures The routine is being recorded without being performed, or the standard is wrong Look at the quality of the check, and review the behaviour statement
Records show the routine and failures fall The design is doing its job Keep it, and check again at the next review cycle

Best practice: report honestly, including what you cannot know. A firm will want a simple answer. Give the evidence you planned, say what it can and cannot show, and do not claim a change caused by training when other things changed at the same time: a new tool, a revised policy, an incident everyone heard about. Your report is subject to the same standard the session teaches: say where each claim comes from and how far it goes.

Remember this

  • The evidence is planned when the behaviour is written; if you cannot say what you would see, the behaviour is not specific enough.
  • Satisfaction and recall are not evidence of behaviour.
  • Plan evidence in the session, soon after and later, preferring records the work already produces.
  • Count what records show, not what they contain; agree the plan with whoever owns the data and tell participants.
  • Decide in advance what result would change the design, and report what the evidence can and cannot show.

Legal example

The fictional firm Oakhollow Braithe LLP asks its knowledge management director, Caspian Aduke-Lindqvist, to design an AI verification session for its corporate associates. His behaviour statement reads: "When an AI-assisted draft contains any cited authority and is about to go to a supervising lawyer, the associate will open each authority in the firm's research service, compare the cited passage with the draft, and record the check using the verification field on the document management system, instead of relying on the citation looking correct." Before building the session he writes the evidence plan. In the session, participants complete a practice brief containing four authorities, one labelled in his notes as (a fictional citation invented for this lesson; it does not exist), and he collects the verification notes anonymously and reviews them during the break. Soon after, he asks the firm's risk manager, who already has authority to see document management reports, to report monthly in aggregate how many AI-flagged drafts in the corporate group carry a completed verification field. He never sees a file. Later, he asks three supervising partners for a short monthly note on whether they are still finding unchecked authorities. The general counsel approves the plan, and Caspian tells participants at the start of the session exactly what will be reported and that nothing is reported by name. He writes down in advance what he expects and what he would do if it did not appear. Eight weeks later, the in-session notes were complete for almost every participant, but completed verification fields on real drafts barely rose. The supervising partners report no change. Caspian does not re-run the session. He sits with two associates and finds that the verification field is three clicks deep and invisible on the laptop view. The fix goes to IT. The partnership receives a report that says what the evidence shows, what it cannot show, and that the obstacle was the system rather than the associates.

Responsible-use warning

Never present a satisfaction score, an attendance list or an end-of-session quiz as evidence that behaviour changed. Never gather evidence by reading client files or monitoring individuals without a clear policy basis, the agreement of whoever owns the data, and telling participants what will be examined. And when the evidence shows no change, look first at the design, the tools and the policy before drawing any conclusion about the people you trained.

Human-review requirement

A licensed attorney, usually the firm's general counsel or risk partner, approves the evidence plan before the session and owns decisions about access to matter records. The reviewer confirms that evidence is drawn from records the work already creates, that it is reported in aggregate by someone authorised to see it, that no client information is disclosed to anyone without a need to see it under the jurisdiction's adopted confidentiality rule and firm policy, that participants are told what will be examined, and that the report does not overclaim what the evidence shows.

Practical exercise

Write the evidence plan for the session you have been designing through this module. Use fictional participants and fictional records.

  1. Copy your behaviour statement onto the top of the page. If you cannot say what anyone would see when it happens, rewrite it now.
  2. Describe the in-session evidence: what the practice task produces, how you will collect it, and when you will look at it.
  3. Describe the soon-after evidence: which record the work already creates that would show the routine, who is authorised to see it, and how it will be reported in aggregate.
  4. Describe the later evidence: how you will learn whether the failure the behaviour prevents is still occurring.
  5. List every point at which the plan could involve client information, and redesign each so that nobody sees contents they do not need.
  6. Write the two sentences you will say to participants about what will be examined and how.
  7. Complete a table of at least three possible results, with the most likely meaning and your next step for each.
  8. Name the person whose approval the plan needs before the session.
  9. Set the dates on which you will look at the soon-after and later evidence, and put them in your calendar now.

Knowledge check

  1. Why are satisfaction scores and end-of-session quizzes weak evidence of behaviour change?

    Answer guidance: A satisfaction score shows participants enjoyed the session and a quiz shows immediate recall; neither shows that anyone recognises the trigger in real work or performs the routine at their desk weeks later.

  2. What are the three points at which a session's evidence should be planned?

    Answer guidance: In the session (what participants produce in the practice), soon after (whether the routine appears in records the work already creates), and later (whether the failure the behaviour prevents is still occurring).

  3. How can evidence about a verification routine be gathered without the instructor seeing client material?

    Answer guidance: By counting what existing records show, such as whether a verification field is completed, reported in aggregate by someone already authorised to see the system, under a plan agreed with the firm's general counsel or risk function and explained to participants.

  4. Records show a routine being recorded, but supervisors still find unchecked authorities. What is the likely meaning?

    Answer guidance: The routine is being recorded without being properly performed, or the standard in the behaviour statement is wrong; the next step is to look at the quality of the check and review the statement rather than simply re-running the session.

Sources and further reading