Lesson objective
Explain automation complacency and automation bias as they apply to high-volume accounting review, and design review controls that do not depend on reviewers simply trying harder.
Plain-language explanation
Why this matters now
The reviewers in an AI-assisted operation are not careless people. They are experienced accountants working through a queue during the close or busy season, with a tool that is usually right. That is exactly the setting in which decades of research on automated decision aids predicts that checking will slip.
Research: Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors, vol. 52, issue 3 (June 2010), is a review that builds an integrated model from earlier studies. Three of its conclusions matter for accounting operations:
- "Automation complacency occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention."
- Automation complacency "is found in both naive and expert participants and cannot be overcome with simple practice."
- "Automation bias results in making both omission and commission errors when decision aids are imperfect," and it "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams."
Research: Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, "Automation bias: a systematic review of frequency, effect mediators, and mitigators," Journal of the American Medical Informatics Association, vol. 19, issue 1 (2012), defines automation bias as "the tendency to over-rely on automation." Of 13,821 papers retrieved, 74 met the inclusion criteria. Among the factors that made it worse were "workload, task complexity, and time constraint."
Both are pre-LLM studies from fields such as aviation, process control and clinical decision support. They apply to accounting by analogy. Still, a close team with a deadline, a mixed queue of tasks and a mostly reliable tool matches the conditions they describe closely.
Professional guidance (voluntary framework): NIST's Generative Artificial Intelligence Profile (NIST AI 600-1, July 2024) names Human-AI Configuration as a generative AI risk, including humans "experiencing algorithmic aversion, automation bias, over-reliance, or emotional entanglement" with these systems. It describes automation bias as "excessive deference to automated systems" and says it "can exacerbate other risks of GAI, such as risks of confabulation." The profile is voluntary and not written for accountants.
A question to sit with
Your best reviewer found almost nothing wrong in last month's AI-prepared reconciliations. Does that tell you the outputs were right, or that the review had stopped looking?
What goes wrong
Omission errors. The tool fails to flag something, and the reviewer, relying on the tool to raise problems, never looks. A duplicate vendor payment that the matching tool did not mark as unusual sails through because nobody scans unmarked items.
Commission errors. The tool suggests something wrong, and the reviewer accepts it. A suggested account for an invoice is plausible, the reviewer is on item 140 of 300, and the suggestion is approved.
Reviewing only what the tool marks as uncertain. Research (observational): Choi and Xie (Journal of Accounting Research, 2026) found that accountants on the platform they studied "selectively intervene when AI confidence scores are low." That is sensible use of reviewer time. The risk is that a confidently wrong output, the kind that matters most, lands in the pile nobody opens. Their framed field experiment also found that AI assistance improved classification accuracy on average, but "reliance on non-consensus AI recommendations can increase the risk of error." Those findings come from one platform and its small business clients.
Busy season as the default state. Workload and time pressure make over-reliance worse, and the peak weeks are when volume is highest. The controls weaken exactly when they carry the most weight.
Team diffusion. The preparer assumes the reviewer will catch it. The reviewer assumes the preparer traced it. Parasuraman and Manzey note automation bias can affect teams as well as individuals.
The sign-off as a ritual. A tick in a review column records that someone opened the item, not what they compared it with.
The practical answer
Design the review so that the right behaviour happens even when people are tired. Goddard and colleagues list mitigators that included "training and emphasizing user accountability" and design factors such as where advice appears on screen, updated confidence levels, and "the provision of information versus recommendation." The table below turns those findings into accounting review controls. The mapping is this programme's recommendation, drawn by analogy from research outside accounting.
| Condition that feeds over-reliance | Review control | What it changes |
|---|---|---|
| Reviewer relies on the tool to raise problems | Random sample of unflagged, high-confidence items traced to source each cycle | Omission errors in the "clean" pile can be found |
| Suggestion shown as an answer | Show the source evidence first and the suggestion second, or show evidence without a suggestion for a sample | Reviewer judges the evidence, not the proposal |
| Tool and reviewer disagree, or the tool departs from normal treatment | Reviewer records their own treatment before seeing the suggestion on those items | Reduces uncritical adoption of non-consensus suggestions |
| Workload and time pressure | A maximum items-per-reviewer-per-day during peak, with overflow routed to another reviewer or the next day | Keeps review depth from silently collapsing |
| Diffuse responsibility | Each review entry names the reviewer and the source document compared | Makes accountability specific |
| Ticks without evidence | Review field requires a source reference, not just a checkbox | The record shows what was compared |
Sample the confident pile on purpose. AI Coalition Network recommendation: every review cycle, pull a fixed number of items the tool marked as confident and unflagged, and trace them fully to source. Record what the sample found. If it finds nothing for months, keep sampling: finding nothing is evidence only while the sample keeps running. Module 13 sets sample sizes and error categories.
Separate information from recommendation where it counts. For items with judgement in them, such as an unusual account classification, an accrual or a revenue item, present the source document and the relevant facts first. Let the reviewer decide, then compare with the tool's suggestion. Differences are worth a note either way.
Protect review capacity in peak weeks. Put the capacity limit in the procedure, with a named person who decides what happens to overflow. A queue that grows past what reviewers can examine is a stop condition, which Module 14 covers.
Watch the review, not just the output. Look for signs that checking has thinned: review times falling steadily while volume rises, no corrections recorded for long periods, identical review notes. These are prompts to look, not proof of failure.
Remember this
- Complacency and automation bias affect experts as well as beginners, and get worse under workload and time pressure.
- Omission errors hide in the items the tool did not flag; sample that pile deliberately.
- Reviewing only low-confidence items leaves confidently wrong outputs unexamined.
- Build controls that work when people are tired: evidence before suggestion, capacity limits, named source references.
- Training helps, but instructions alone do not prevent over-reliance.
Accounting example
Oakridge Mercer LLP is a fictional firm running AI-suggested general ledger coding for 45 fictional client entities. The tool codes each vendor invoice and gives a confidence score. The procedure routes items scored below 80 to a reviewer and posts the rest after a daily batch review by a senior. In a quiet month the batch review is thorough. In January the queue triples. Senior accountant Jonah Ellis reviews about 300 items a day, and his review time per batch falls from 50 minutes to 15. In February, a client's controller asks why repairs expense is so low. The team finds that invoices from a new equipment supplier were coded to fixed assets with scores above 90, so none reached a reviewer. Eleven invoices totalling $38,500 should have been expensed, according to the client's capitalisation policy. Manager Priya Nandakumar redesigns the review rather than asking Jonah to try harder. Each day, 20 items scored above 80 are drawn at random and traced to the invoice and the capitalisation policy. Any item from a vendor with no coding history goes to a reviewer regardless of score. During January to April, no reviewer handles more than 150 items a day; overflow moves to a second senior named in the procedure. Reviewers record the invoice number they compared, not a tick. In the first month of the new design, the random sample finds 2 wrong codings among 440 sampled items, both on new vendors, and the vendor rule catches 6 more before posting. Priya records the findings for the quality assurance file.
Responsible-use warning
Do not treat a quiet review log as evidence that outputs are correct. A review that finds nothing may be a review that stopped looking, especially in the busiest weeks. And do not route only low-confidence items to people: a confidence score is the tool's view of itself, and a confidently wrong output is the one most likely to reach the client. Keep a random sample of confident, unflagged items under full review every cycle.
Human-review requirement
The qualified professional who owns the output, whether the licensed CPA on an attest engagement, the signing tax return preparer or the responsible controller, owns the review design as well as the sign-off. They check that the sample of confident items was actually drawn and traced, that capacity limits held in peak weeks, and that review entries name the source compared. They decide what happens when the sample finds errors. A reviewer's tick is not evidence of review; the recorded comparison is.
Practical exercise
Use fictional data only.
- Describe a fictional AI-assisted review queue in your area: what the tool suggests, how items reach a reviewer, and roughly how many items a day arrive in normal and peak weeks.
- Identify where an omission error could hide: which items does no person ever look at?
- Identify where a commission error could happen: where does a reviewer accept a suggestion without seeing the source?
- Write a sampling rule for confident, unflagged items: how many items per cycle, how they are selected, and what "traced to source" means for this task.
- Pick one judgement-heavy item type and describe how the reviewer would see the evidence before the suggestion.
- Set a daily capacity limit per reviewer for peak weeks, and name the role who decides what happens to overflow.
- Rewrite the review field so it records the source compared, not just a tick.
- List three signals you would watch for that checking has thinned, and state what you would do when one appears.
- Run a short thought test: imagine the tool is confidently wrong on one item type for a whole month. Using your redesigned queue, write down which control would find the problem first, roughly how many days it would take, and who would be told.
- Record your design in the workbook and note which parts are your own recommendation rather than a firm rule.
Knowledge check
-
What is the difference between an omission error and a commission error in AI-assisted review?
Answer guidance: An omission error is missing a problem because the tool did not flag it and the reviewer relied on the tool to raise problems. A commission error is acting on a wrong suggestion the tool made, such as approving a wrong account code.
-
Why is reviewing only low-confidence items risky, and what does Choi and Xie's study add?
Answer guidance: A confidently wrong output lands in the unreviewed pile. Choi and Xie observed accountants selectively intervening when confidence scores were low, and found reliance on non-consensus AI recommendations can increase error risk, though their findings come from one platform.
-
What did Parasuraman and Manzey conclude about training and automation bias, and how should that be read?
Answer guidance: They concluded automation bias occurs in naive and expert participants and cannot be prevented by training or instructions. That does not mean training is worthless; it means instructions alone are not enough, so review design must carry the control.
-
Name three review controls that work under workload without relying on reviewers trying harder.
Answer guidance: Any three of: a random sample of confident, unflagged items traced to source; showing evidence before the suggestion; a daily capacity limit with a named overflow decision; review entries that record the source compared; recording the reviewer's own treatment before seeing a suggestion on non-consensus items.
Sources and further reading
- Raja Parasuraman and Dietrich H. Manzey; Human Factors (Human Factors and Ergonomics Society) — Complacency and Bias in Human Use of Automation: An Attentional Integration (2010-06-01). Accessed 2026-09-15.
- Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt; Journal of the American Medical Informatics Association — Automation bias: a systematic review of frequency, effect mediators, and mitigators (2012-01-01). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- Jung Ho Choi (Stanford Graduate School of Business) and Chloe L. Xie (MIT Sloan School of Management), Journal of Accounting Research — Human + AI in Accounting: Early Evidence from the Field (2026-04-16). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.