Lesson objective

Distinguish the refusal failure from the unread-acceptance failure, and state honestly what the automation-bias reviews and the jagged-frontier experiment do and do not support about either.

Plain-language explanation

Why this matters now

A practice AI session is aimed at two wrong outcomes on opposite sides of the work, and steering hard away from one lands a room in the other.

Refusal is the outcome nobody reports. The registration clerk retypes a demographics block the tool already assembled, because retyping feels safer than reading. The clinician glances at the ambient draft, opens a blank note and dictates from the start, so the evening ends when it always ended. It reaches no patient and no incident log. It surfaces, if at all, as a practice that paid for something and changed nothing.

Unread acceptance is the outcome that leaves a name on it. A drafted portal reply goes out carrying an interval nobody compared with the chart. An ambient-generated note is signed at 19:40 alongside thirteen others, and that signature attests a clinician read it. A prior authorisation letter states a rationale more confidently than the documentation behind it. Each lands in a record, a payer file or a patient's inbox.

They surface differently, which is why an instructor preventing the visible one keeps manufacturing the invisible one. Half an hour on hazards sends a room back to the paper pad; half an hour on hours saved sends it back to the signature. Holding both is the work of the first minute, and the honest sentence about the second is the one nobody wants to say aloud: telling people to check does not reliably make them check.

Two pieces of work sit behind that sentence, each with limits.

Research (not binding): Parasuraman and Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration", Human Factors, vol. 52, issue 3, pp. 381-410, June 2010 — a review and integrated theoretical model. The complacency they describe arises "under conditions of multiple-task load", where manual work competes for the operator's attention, and they report it "is found in both naive and expert participants". Of automation bias they write that it "cannot be prevented by training or instructions", reaching individuals and teams alike, and that imperfect decision aids draw "both omission and commission errors" from their users.

Research (not binding): Goddard, Roudsari and Wyatt, "Automation bias: a systematic review of frequency, effect mediators, and mitigators", Journal of the American Medical Informatics Association, vol. 19, issue 1, pp. 121-127, 2012 — built from 74 papers that met inclusion out of 13,821 retrieved, defining automation bias as "the tendency to over-rely on automation". Its mediators include user factors such as cognitive style and task-specific experience, trust and confidence, and the environmental trio "workload, task complexity, and time constraint". Its mitigators include "training and emphasizing user accountability" and design choices such as "the provision of information versus recommendation".

Both examined automated decision aids, in fields such as aviation, process control and clinical decision support, and both predate large language models. Neither studied teaching. No session, curriculum or instructor was measured by either, and a lesson borrowing them owes a room that sentence first.

A question to sit with

It is 19:40 on a Tuesday. Fourteen notes wait on a signature and the portal shows thirty-one unread messages, nine with drafts under them. Your slide said to read the draft against the chart before signing. Between that sentence and the fourteenth signature, what else stands there?

What goes wrong

1. One sentence is asked to carry a system. "Always verify" is true, and the least load-bearing thing said all session. Parasuraman and Manzey report that training or instructions did not prevent automation bias.

2. Seniority mistaken for exemption. A physician of twenty-five years says the material cannot be about her. Both reviews report the effect among expert participants as well as novices, and reading an AI-assembled chart summary is a task she has done for six weeks. Task-specific experience is named as a mediator, not as protection.

3. The finding bent backwards, then forwards. A partner concludes the practice should stop paying for training and buy a software control instead; Goddard and colleagues put training with emphasis on user accountability among the mitigators, so the research supports no such move. Elsewhere a percentage appears for how often support staff accept a draft unread. No such number exists here, and neither review watched practice staff use a drafting tool.

4. Only the wrong line rehearsed, never the missing one. Learners practise catching a statement that contradicts the chart, never noticing a medication a summary failed to carry over. Omission and commission both appear in the research, and the omission surfaces later, at a refill or a denial, when nobody connects it to a draft.

5. The chatbot already in the room. Guidance (voluntary, independent nonprofit): ECRI ranked "Misuse of AI chatbots in healthcare" first in its Top 10 Health Technology Hazards for 2026, press release dated 21 January 2026. ECRI says such chatbots "are not regulated as medical devices nor validated for healthcare purposes" while clinicians, patients and health care personnel use them anyway. Its experts report incorrect diagnoses, unnecessary testing and "invented body parts"; in one ECRI test a chatbot placed an electrosurgical return electrode over the shoulder blade, which would risk burns. ECRI recommends verification with a knowledgeable source, governance committees, clinician AI training and regular audits. Plan for a room where somebody has already asked a chatbot a clinical question.

6. A verified silence heard as permission. Someone asks what the state medical board requires. As verified on 2026-09-15, this register holds no state medical board rule on AI use, no disciplinary case, no AMA Code of Medical Ethics opinion on AI, and no regulator requiring a record of AI use or a retention period; AMA's position on augmented intelligence is advocacy, binding nobody present. Say the silence was checked and dated, then that a silence is not permission.

The practical answer

What an instructor may say, and may not say.

You may say You may not say
Automation bias is "the tendency to over-rely on automation" (Goddard and colleagues, 2012). "Support staff accept AI drafts unread 40% of the time." No such figure exists here.
Workload, task complexity and time constraint are the named mediators. "A double-booked afternoon doubles the error rate."
It concerns decision aids in other fields and predates large language models. "This was measured on medical assistants using drafting tools."
ECRI ranked chatbot misuse first among its 2026 hazards. "ECRI measured how often chatbots are wrong." It published no such rate.
As verified on 2026-09-15, this register carries no independent error rate for any commercial ambient documentation product. "Ambient documentation is right nine times in ten."

Best practice: rehearse the missing line as hard as the wrong one. A commission error is acting on something a draft got wrong — a fictional reminder gives a fictional patient a follow-up interval the chart does not support, and the front desk sends it. An omission error is missing what a draft never raised — a fictional chart summary carries two active medications and drops a third that sits only on the intake form, and the reader, handed a tidy paragraph, never opens it. Build a round of each on fictional material.

Best practice: run the exercise at the hour the failure happens. Multiple-task load, time constraint and task complexity are the mediators these reviews name, and they describe a medical practice: rooms turning over, a prior authorisation deadline, a portal queue worked between patients. An exercise in a quiet break room at 12:20 rehearses conditions no learner meets. Where they cannot be changed, move the check to the point of least load.

Best practice: fence the jagged-frontier experiment first. Research (not binding): Dell'Acqua and colleagues, "Navigating the Jagged Technological Frontier", Organization Science, vol. 37, issue 2, March 2026 — a preregistered experiment with 758 knowledge workers, consultants at Boston Consulting Group, assigned at random to no AI, GPT-4, or GPT-4 with a prompt engineering overview. Across 18 tasks inside the AI capability frontier, those using AI completed 12.2% more tasks, 25.1% more quickly, with significantly improved quality; on a complex task outside it, subjects using AI "were 19% less likely to produce correct solutions compared with those without AI". Consultants, GPT-4 as of 2023, no clinician and no teaching in it. Its one honest use is to explain why "it was fine last time" settles nothing, and why a practice decides task by task. Not a medical figure.

Best practice: put the accountability in the room, on paper. Emphasising user accountability is a named mitigator, and in a practice it has a physical shape: a named person reviews an output against its source before it leaves, and the review is recorded where it can be found. Bring the practice's own review record in rather than a specimen, and have every learner complete one.

AI Coalition Network recommendation: treat the session as one component and name the others. A session moves what a person knows how to do. The written review rule, the field holding the reviewer's name, the tool's configuration and the minutes allowed for a check carry the rest, and none belongs to the instructor. Finish every design by writing what else must be true at the workstation, in the exam room and in billing, then hand each line to its owner. This is our educational standard, not a rule from any board or association.

Remember this

  • Refusal hides in work not done; unread acceptance surfaces in a record, a claim or a patient's inbox, with a name on it.
  • State it at its own strength: the effect appeared among experts as well as novices, training or instructions alone did not prevent it, and training with accountability is a mitigator.
  • Both reviews concern decision aids from other fields before large language models, and neither measured teaching; say so before quoting either.
  • Rehearse the omission as well as the contradiction, on fictional patients and charts.
  • ECRI ranked chatbot misuse first among its 2026 hazards and published no error rate; a dated silence in the register is never permission.

Medical practice example

Priya Nkemelu, a fictional instructor, is asked by the fictional Brightfield Pediatrics to run a forty-minute session after two near misses. In the first, a drafted portal reply told a fictional parent that a fictional child's follow-up was in three months when the chart said one; a medical assistant sent it unread at the end of a Friday. In the second, a drafted chart summary omitted a current medication that appeared on the intake form, and the clinician signing the note did not open the form. Priya builds the session around those two shapes rather than around the tool. She opens by naming the two failures and says plainly that the practice has now produced one of each. She then puts the automation-bias material up with its limits attached: a review of decision aids in other settings, published 2010, and a healthcare systematic review published 2012 in which 74 of 13,821 retrieved papers met inclusion, both before large language models, neither a study of teaching. She quotes the finding that automation bias occurred in naive and expert participants and was not prevented by training or instructions, and immediately quotes the mitigators, including training with emphasis on user accountability, so nobody leaves believing training is pointless. She runs two timed rounds on fictional material. Round one is a drafted reminder containing a wrong interval, which nine of eleven learners catch. Round two is a fictional chart summary missing a medication, which three of eleven catch. Priya reports both results as what happened in that room on that afternoon and as nothing else. In the debrief the practice manager, Wendell Achebe, notes that both rounds were run at four in the afternoon deliberately, because that is when the queue is cleared. Priya closes with the list of what else must be true: a written review rule, a named reviewer recorded on the portal task, and four minutes allowed for the check. She hands the list to Wendell rather than to the room, because he is the person who can change any of it.

Responsible-use warning

Never tell a room that the automation-bias research proves training works, and never tell them it proves training fails. Never present either review as a study of teaching, of practice staff, or of large language models. Never quote an over-reliance rate: none exists in these sources. Never present the Boston Consulting Group experiment as a medical figure, and never attribute an error rate to ECRI, which published none.

Human-review requirement

The instructor owns every research statement made in the room and every figure on a slide. Before delivery, a second reviewer checks each research claim word by word against the published wording, confirms the setting, date and pre-LLM limits appear beside the claim, and confirms that no teaching-effectiveness claim has been drawn from any of it. The practice owner decides what the practice will change as a result, and the named privacy official confirms every exercise item is fictional. The instructor advises; the practice decides.

Practical exercise

Build the two-failures segment of a fictional session, with the research stated at exactly the strength it will bear.

  1. Write a fictional near miss of each kind at a fictional practice: one commission error, where a drafted output stated something wrong and a person acted on it; one omission error, where a drafted output left something out and nobody went back to the source.
  2. For each, name the role that produced it, the named owner of that output, and the condition at that moment: load, time pressure or complexity.
  3. Write the two sentences you will use to introduce the automation-bias reviews, each carrying the setting, the publication year and the fact that they predate large language models.
  4. Write the sentence you will use for the mitigators, in a form that cannot be heard as "training fixes it".
  5. Write, word for word, your answer to a learner who says the research means training is a waste of the practice's money.
  6. Write your answer to a clinician who says twenty-five years of experience makes the finding irrelevant to them.
  7. Design two timed rounds on fictional material, one commission and one omission, each no longer than four minutes, and write the debrief question for each.
  8. Write one sentence saying what you will and will not claim about the results of those rounds.
  9. Write the two sentences you will use for the ECRI chatbot hazard, including that ECRI published no error rate, and the sentence you will use when someone admits they asked a general chatbot a clinical question last week.
  10. Write the one honest use of the Boston Consulting Group experiment in a medical session, and one sentence stating what it is not.
  11. List everything besides the session that must be true at the workstation for the behaviour to survive, and name the person who owns each item.

Knowledge check

  1. What are the two failures a practice AI session must hold at once?

    Answer guidance: Refusal, where the tool is not used and the work is late or undone, and unread acceptance, where a drafted output reaches a patient, a record or a claim without being read against the source.

  2. State the automation-bias finding about training at exactly the strength the sources support.

    Answer guidance: Automation bias occurred in naive and expert participants and was not prevented by training or instructions alone, while training with emphasis on user accountability appears among the named mitigators.

  3. Why must both automation-bias reviews be labelled before they are used in a medical session?

    Answer guidance: Because they concern automated decision aids in settings such as aviation, process control and clinical decision support, predate large language models, and studied no teaching, so they reach practice work only by analogy.

  4. What may an instructor say about ECRI's 2026 hazard list?

    Answer guidance: That ECRI ranked misuse of AI chatbots first, said such chatbots are not regulated as medical devices nor validated for healthcare purposes, and recommended verification with a knowledgeable source, governance, training and regular audits.

Sources and further reading