Lesson objective
Explain what the automation bias research does and does not support, and build a session in which accountability and working conditions carry weight that instruction alone cannot.
Plain-language explanation
Why this matters now
Almost every AI session for insurance professionals closes the same way: a slide reading "Always check the output." Nobody in the room disagrees. Then, three weeks later, on the Thursday before a large renewal, an account manager accepts a coverage summary the tool produced because it looked right and the tool has been right before.
This programme holds no research on how adults learn, so it can say nothing about whether your slide was memorable. It does hold research on how people use automated aids, and that research speaks directly to the closing slide.
Research (not binding): Parasuraman and Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration", Human Factors, vol. 52, issue 3, pp. 381-410, June 2010. A review and integrated theoretical model. The authors report that automation complacency "occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention", and that it "is found in both naive and expert participants and cannot be overcome with simple practice." They report that "Automation bias results in making both omission and commission errors when decision aids are imperfect", and that automation bias "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams."
Research (not binding): Goddard, Roudsari and Wyatt, "Automation bias: a systematic review of frequency, effect mediators, and mitigators", Journal of the American Medical Informatics Association, vol. 19, issue 1, pp. 121-127, 2012. It defines automation bias as "the tendency to over-rely on automation." Of 13,821 retrieved papers, 74 met the inclusion criteria. Mediators included user factors such as cognitive style and task-specific experience, trust and confidence, and environmental factors: "workload, task complexity, and time constraint." Mitigators included "training and emphasizing user accountability" and design factors such as the position of advice on screen, updated confidence levels, and "the provision of information versus recommendation."
Both concern automated decision aids in fields such as aviation, process control and clinical decision support. Both predate large language models. They reach insurance agency work only by analogy, and a lesson must say so. Neither says training is worthless. The finding is that training or instructions alone did not prevent automation bias.
The conditions the reviews name are the conditions of an agency. Multiple-task load describes an account manager holding four renewals, a certificate request and a claim follow-up. Time constraint describes the day before an effective date. Workload describes January in a Medicare office. Nothing in this register measures the effect in insurance work, and you must not imply that anything does.
A question to sit with
If a warning slide could prevent over-reliance, the review literature would have found it among the mitigators. Besides your words, what will be in place at that desk on the Thursday before the renewal when a confident, wrong coverage summary appears?
What goes wrong
1. The instruction carries the whole load. "Always verify" is true and necessary. Alone, it is an instruction, and the reviews report that automation bias was not prevented by training or instructions alone.
2. "Our producers are veterans, so this does not apply." Both reviews report the effect in expert participants as well as naive ones. Task-specific experience is a mediator, not immunity, and a twenty-year producer reading an AI coverage summary is doing a new task.
3. "They will get the hang of it." Parasuraman and Manzey report that complacency cannot be overcome with simple practice. Practice is still worth doing. It is not a cure.
4. The research read backwards. A consultant tells an agency the studies prove training is money wasted. They do not. Goddard and colleagues list training and emphasising user accountability among the mitigators.
5. The conditions ignored. The session runs in a quiet conference room, one task at a time, with unlimited time. It never meets the multiple-task load the reviews name, so it rehearses a situation the learner will not be in.
6. Only one kind of error practised. Learners practise catching a wrong statement the tool made. They never practise noticing what the tool left out. Automation bias produces both omission and commission errors.
The practical answer
What an instructor may say, and may not say.
| You may say | You may not say |
|---|---|
| Automation bias is "the tendency to over-rely on automation" (Goddard and colleagues, 2012). | "Producers over-rely on AI 40% of the time." No such figure exists in these sources. |
| It appeared in naive and expert participants, and training or instructions alone did not prevent it. | "Training does not work." |
| Workload, task complexity and time constraint were named mediators. | "Renewal season doubles the error rate." |
| Training and emphasising user accountability were named mitigators. | "Accountability eliminates automation bias." |
| Complacency occurred under multiple-task load and was not overcome by simple practice. | "Experienced producers are immune." |
| The research is pre-LLM, from other fields, and applies here by analogy. | "This research was done on agency AI tools." |
Best practice: teach both kinds of error with insurance material. An omission error is missing what the aid never raised: a fictional AI summary of a fictional commercial policy lists the general liability limits and says nothing about a fictional exclusion endorsement that is on the declarations, and the reviewer, reading a tidy summary, does not go looking. A commission error is acting on what the aid said when it was wrong: the same summary states a fictional water backup limit of $25,000 when the declarations say $10,000, and the account manager repeats it in an email to the client. A session that shows only the second teaches half the problem. In insurance, the first is often the more expensive, because a coverage gap surfaces at a claim.
Best practice: treat any confidence signal as information, not permission. Where a tool marks part of an extraction as uncertain, the sensible habit is to review every uncertain item and a sample of the confident ones, with the sample size set by the agency's written rule rather than by the instructor. Nothing in this register measures how often such signals are right.
Best practice: make accountability visible during the session. Goddard and colleagues name emphasising user accountability as a mitigator. In an agency that has a concrete shape: a named licensed person reviews the output and the review is recorded. Show learners the actual review record the agency uses, and have every learner complete one during the hour on a fictional item.
Best practice: rehearse under something like real conditions. If time constraint and multiple-task load are named mediators, an untimed exercise does not rehearse the moment that matters. Run one timed round with a realistic volume of fictional documents, then debrief which items were accepted too fast. Describe the result as what happened in that room, never as a measurement.
AI Coalition Network recommendation: design the session as one part of a system. The session carries part of the weight. The agency's review rule, the record of who reviewed what, the tool's settings and the minutes allowed for review carry the rest. When you finish a design, write the list of what else must be true at the desk and hand it to the person who owns each item.
Remember this
- Automation bias appeared in naive and expert participants and was not prevented by training or instructions alone.
- Training and emphasising user accountability are named mitigators; workload, task complexity and time constraint are named mediators.
- Teach both omission and commission errors, and use insurance documents to do it.
- A confidence signal is information; review the uncertain items and sample the confident ones under the agency's rule.
- This research is pre-LLM and from other fields, and reaches agency work only by analogy.
Insurance example
Ines Brightwater, a fictional commercial lines manager and trainer at the fictional Larkspur Benefits Group, inherits a session whose last slide reads "AI gets things wrong. Always check." She rebuilds it. First she adds a slide quoting Parasuraman and Manzey directly: automation bias occurred in naive and expert participants and could not be prevented by training or instructions, with the caveat printed underneath that the research predates large language models and comes from other fields. Then she builds a timed round. Each learner receives a fictional AI-produced summary of a fictional commercial package policy for the fictional insured Cobblewright Machining, written from a fictional declarations page and a fictional endorsement schedule. The summary contains 18 statements. Three are wrong: a general liability aggregate stated as $4,000,000 when the declarations say $2,000,000, a deductible stated as $2,500 when the schedule says $5,000, and an equipment breakdown limit that appears nowhere in the documents. The summary also omits a fictional exclusion endorsement listed on the declarations, which is the planted omission. Three wrong statements plus one omission makes four planted problems in eighteen statements. Learners have fourteen minutes and are interrupted twice by a fictional certificate request handed across the table, which is how Ines rehearses multiple-task load. In the debrief most learners found the aggregate error, fewer found the deductible, and only two mentioned the missing endorsement. Ines says plainly that this describes the fifteen people in the room that morning and is not a measurement of anything. She then shows the agency's real review record, which names the licensed reviewer and the items checked, and every learner completes one for the fictional file. Afterwards she writes to the operations manager who owns the tool: the tool shows its answer without the document text beside it, and she recommends he look at whether it can show both. The decision stays with him.
Responsible-use warning
Never let a closing instruction carry the weight of a session, and never tell an agency the research proves training is pointless. Parasuraman and Manzey report that training or instructions alone did not prevent automation bias, and Goddard and colleagues list training and emphasising user accountability among the mitigators. Never present this pre-LLM research from other fields as a study of insurance AI tools, never report what happened in your room as a measurement, and never teach that a confident-looking output has been checked.
Human-review requirement
The instructor owns the accuracy of every statement about the research. Before delivery, a second reviewer checks each quotation against the published paper, confirms each is labelled as pre-LLM research from other fields applying by analogy, and confirms no rate or percentage has been invented. The agency owner of the review rule confirms that the review record used in the session is the one the agency actually uses and that the sample size taught matches the written rule. The named licensed reviewer on each real file, not the instructor, remains accountable for accepting or rejecting AI output in customer work.
Practical exercise
Design a timed round that rehearses both kinds of error. Use a fictional insured, fictional policy documents and a fictional tool throughout.
- Write a fictional declarations page and endorsement schedule for a fictional commercial or personal lines policy, with at least twelve stated values: limits, deductibles, forms and endorsement titles. Mark every form and endorsement number as fictional.
- Write a fictional AI-produced summary of that policy containing fifteen to twenty statements. Count them and record the total.
- Plant at least two commission errors: statements that contradict the documents. Record exactly where each one is and what the document actually says.
- Plant at least one omission: something on the declarations the summary never mentions. Record it, and write one sentence on how it could surface at a claim.
- Confirm your planted problems and your statement count add up, and write the addition down.
- Set a time limit that reflects a working day rather than a classroom, and add one realistic interruption. Write one sentence explaining the choice as instructional practice, not as a research finding.
- Write the review habit you will teach: which items are always checked against the document, and how a sample of the rest is chosen under the agency rule.
- Draft the review record the learners will complete, naming the licensed reviewer, the items checked against the source documents, and the date.
- Write two sentences you will say about Parasuraman and Manzey and about Goddard and colleagues, each carrying its caveat.
- Write your debrief questions, including one about the planted omission, and make sure none of them asks a learner to describe a real customer file.
Knowledge check
-
What did Parasuraman and Manzey report about automation bias and training?
Answer guidance: That automation bias occurs in naive and expert participants, cannot be prevented by training or instructions, and can affect decision making by individuals and by teams.
-
Which mediators and mitigators did Goddard and colleagues name?
Answer guidance: Environmental mediators included workload, task complexity and time constraint; mitigators included training and emphasising user accountability, plus design factors such as how advice is presented.
-
What is the difference between an omission error and a commission error in a policy summary?
Answer guidance: An omission error is failing to notice something the summary never raised, such as an endorsement on the declarations; a commission error is acting on a statement the summary got wrong.
-
Why must an instructor describe this research as applying by analogy?
Answer guidance: Because both reviews concern automated decision aids in fields such as aviation and clinical decision support and predate large language models, so neither measured insurance agency work.
Sources and further reading
- Raja Parasuraman and Dietrich H. Manzey; Human Factors (Human Factors and Ergonomics Society) — Complacency and Bias in Human Use of Automation: An Attentional Integration (2010-06-01). Accessed 2026-09-15.
- Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt; Journal of the American Medical Informatics Association — Automation bias: a systematic review of frequency, effect mediators, and mitigators (2012-01-01). Accessed 2026-09-15.
- National Association of Insurance Commissioners (NAIC) — Model Bulletin: Use of Artificial Intelligence Systems by Insurers (2023-12-04). Accessed 2026-09-15.
- Independent Insurance Agents & Brokers of America (Big "I"), Agents Council for Technology (ACT) — 2026 Big "I" Agents Council for Technology Tech Trends Report (press release: Two-Thirds of Independent Agents Plan to Increase AI Use This Year) (2026-02-19). Accessed 2026-09-15.
- Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors (Human Factors and Ergonomics Society), 2010
- Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association, 2012