Lesson objective

Describe how multi-task load and automation bias lead agency reviewers to over-rely on AI output, and place approval gates so that a wrong draft cannot become a published post, a sent email or a changed bid without a person.

Plain-language explanation

Why this matters now

Every review step in Level 2 assumed a person with time to look. In an agency operation, the person doing the review is also answering the client, briefing a designer, fixing a broken tracking link and clearing sixty community replies before a launch goes live. A review step that works on a quiet Tuesday can quietly stop working on launch day, and the SOP will still say it happened.

At the same time, AI tools in agencies are moving from drafting to doing. A tool that writes a reply is one thing. A tool that posts the reply, schedules the email, adjusts the bid or publishes the page is another. When the tool acts, the review step is the only thing between a confident mistake and the public.

Research (not binding): Parasuraman and Manzey's review in Human Factors (June 2010) found that "automation complacency occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention." They found that automation bias "results in making both omission and commission errors when decision aids are imperfect." The research concerns automated decision aids in fields such as aviation and process control and predates language models, so it applies to agencies only by analogy. The analogy is close, because a launch day has exactly the conditions the research names.

A question to sit with

Your most experienced social manager approves AI-drafted community replies from a phone during a launch. She has approved several thousand without a problem. The reply tool can now post directly once she taps approve. What would have to be true for you to trust that her next tap means the reply was actually read?

What goes wrong

Omission errors: the flag that never came. The reviewer relies on the tool to raise problems, so a problem the tool does not raise is missed. An AI reply that leaves out the required answer to a customer's safety question, or an email that drops the opt-out line, raises no alarm, and a busy reviewer does not look for what is absent.

Commission errors: following the suggestion. The reviewer acts on what the tool says even when the evidence in front of them says otherwise. An AI-suggested reply that promises a refund the client's policy does not offer gets approved; an AI-drafted post announcing a sale that ended yesterday gets scheduled.

Experience does not protect. Parasuraman and Manzey found automation complacency "in both naive and expert participants" and that it "cannot be overcome with simple practice." They found automation bias "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams." Your best account lead is not immune, and neither is a two-person sign-off.

Confident text makes it worse. Professional guidance (voluntary framework): NIST's Generative AI Profile (AI 600-1, July 2024) lists Human-AI Configuration among its twelve generative AI risks, including humans "experiencing algorithmic aversion, automation bias, over-reliance, or emotional entanglement." It describes automation bias as "excessive deference to automated systems" and says it "can exacerbate other risks of GAI, such as risks of confabulation." A fluent reply invites a quick approval precisely when a wrong promise is buried inside it.

The wrong answer becomes a wrong action. Voluntary community guidance (international): the OWASP Top 10 for LLM Applications 2026 (dated August 4, 2026) lists Misinformation as LLM07. Its project leads note that voters placed Misinformation near the bottom but the incident record placed it near the top, and explain why: "When a model's fluent, confident output drives a decision or a tool call, a wrong answer turns into a wrong action." The 2026 list also moved Excessive Agency up to third place (LLM03). OWASP is a community consensus document, not a regulation, and nothing in it is specific to marketing.

The practical answer

Design the review so it still finds errors on the worst day of the quarter, and place it before any action the tool can take. The controls below are an AI Coalition Network recommendation, not a legal requirement.

Know what each tool can do, not only what it writes. List every AI-assisted step and mark where on this ladder it sits.

Level What the tool does Example in an agency Minimum control
1. Draft Produces text or an image a person copies Drafts ad headlines in a chat window Review against the claims file before use
2. Suggest Proposes a change inside a system Suggests a bid change in a media platform A person accepts each suggestion with the source on screen
3. Act on approval Executes after a person approves Posts a reply once the social manager taps approve Named approver; sampled check that approvals were real
4. Act alone Executes with no person in the loop Sends a triggered email variant it wrote itself Only for content already approved word for word; otherwise move it down a level

Voluntary framework (NIST AI RMF 1.0, January 2023): Appendix C says human-AI configurations can span "from fully autonomous to fully manual" and that human roles in decision making and oversight "need to be clearly defined and differentiated." The ladder above is this programme's way of doing that for agency tools. NIST does not prescribe it, and the framework binds no agency.

Build the system so a fooled model breaks nothing important. The OWASP project leads put it this way: "Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks." For an agency, "nothing important" includes a public claim, a message to a person who opted out, a client's ad budget and a statement to a journalist. Keep each of those behind a person.

Match controls to the research finding.

Research finding What it looks like in an agency Control
Multiple-task load breeds complacency Sixty replies to clear before a launch goes live Cap reviews per person per hour; move overflow to a second reviewer
Omission errors A reply or email missing a required element A short checklist of required elements shown beside every draft
Commission errors Approving a promise the client's policy does not make Show the policy, offer terms or claims file beside the draft
Experts are not immune "She has done this for years" Sample experienced reviewers as often as new ones
Teams are not immune Two people each assuming the other checked One named approver per item, recorded

Sample the pile nobody looks at. Each week, draw outputs at random from those approved without comment and check them against the source. If the sample finds errors the review should have caught, the review has stopped working and the approval count will not tell you. Module 13 builds the full sampling plan and Module 14 the stop conditions.

Remember this

  • Complacency grows under multiple-task load; launch days are the conditions the research describes.
  • Automation bias produces omissions and commissions, affects experts and teams, and is not prevented by training or instructions alone.
  • Know whether each tool drafts, suggests, acts on approval or acts alone, and keep public claims, sends and spend behind a person.
  • Sample approved outputs against the source every week; an approval count measures taps, not review.
  • OWASP's 2026 list is voluntary guidance: a wrong answer that drives a tool call becomes a wrong action.

Marketing agency example

Quillon Bay Social is a fictional agency that runs community management for a fictional outdoor-gear client, Fenmark Trail Co. An assistant drafts replies to comments on the client's posts; a social manager approves each one, and since June the tool posts the reply as soon as approve is tapped. In an ordinary week the team approves about 400 replies. In the launch week for a new tent the volume reaches 1,500, handled by two people who are also running the launch posts. The approval rate stays at 100% and the average time per approval falls from about 50 seconds to about 8. The head of social, Tamsin Oakes, draws a random sample of 30 approved replies from launch week and checks each against the client's approved answer bank and returns policy. She finds 4 with a problem: two promise a free replacement for any storm damage, which the policy does not offer; one tells a customer the tent is "fully waterproof", a claim missing from the claims file; and one answers a question about a child's sleeping bag without the safety wording the client requires. All four were already public. Tamsin does not blame the two managers. She changes the tool so that approval queues the reply for a second, sampled look during launches, caps each reviewer at 120 replies an hour, moves the answer bank to the top of the review screen, and asks the client whether the four replies need correcting in public. She records that the sample, not the approval rate, showed the review had stopped, and she keeps the weekly sample running after the launch ends.

Responsible-use warning

Do not connect an AI tool to a publish, send or spend action and then measure review by approval rate or queue speed. Both rise when review stops. A reviewer approving every reply in eight seconds on launch day may be excellent or may not be reading, and those numbers cannot tell you which. Only a sample checked against the claims file, policy or answer bank shows whether review is finding the errors that exist. Do not answer a fall in review quality with a reminder alone; change the load, the screen and the level at which the tool is allowed to act.

Human-review requirement

The operation owner owns the review design: which actions each tool may take, the load per reviewer, the order of the review screen, the weekly sample, and what happens when the sample finds a missed error. Each named approver owns the replies, posts or sends they approve and should say when the load is too high to review properly. Any reply that touches a product claim, a safety question, a refund or a legal complaint goes to the account lead and, where needed, the client and its counsel, whatever the queue looks like.

Practical exercise

Use fictional material only. No real client, customer comment, post or colleague.

  1. List every AI-assisted step in one workflow your team runs, such as community replies, email variants or media optimisation.
  2. Place each step on the four-level ladder from this lesson: draft, suggest, act on approval, or act alone.
  3. For every step at level 3 or 4, write the action the tool can take and the worst thing that action could put in front of the public or spend.
  4. Write down the review step as the SOP describes it, then as it actually happens on a launch day. Note every difference.
  5. Estimate the review load per person per hour on an ordinary day and on a launch day. Mark where the load exceeds what one person can genuinely check.
  6. Redesign the review screen so the source (claims file, policy, answer bank) comes first and the AI draft second.
  7. Draft a weekly sampling step: how many approved outputs, drawn how, checked against what, by whom and recorded where.
  8. Name who may move a tool down the ladder or pause it when the sample shows review has stopped, and write the one-line record they must make when they do.
  9. List the replies, posts or sends that must always reach the account lead, such as product claims, safety questions, refunds and legal complaints, whatever the queue length.

Knowledge check

  1. Under what condition did Parasuraman and Manzey find automation complacency occurs?

    Answer guidance: Under multiple-task load, when manual tasks compete with the automated task for the operator's attention, which is the condition of a reviewer handling a launch while approving AI drafts.

  2. Distinguish an omission error from a commission error in agency review.

    Answer guidance: An omission error is missing a problem the tool did not flag, such as a missing opt-out line or safety wording. A commission error is acting on a wrong suggestion, such as approving a reply that promises a refund the client does not offer.

  3. Why does OWASP say a wrong answer can become a wrong action?

    Answer guidance: Because when a model's fluent, confident output drives a decision or a tool call, the error is executed rather than just written. A tool that posts, sends or changes bids turns a mistaken draft into a public or financial consequence.

  4. Does the research mean training reviewers about automation bias is useless?

    Answer guidance: No. The finding is that training or instructions alone did not prevent automation bias. Training still has value, but designed controls such as sampling, source-first screens, load caps and approval gates must carry the weight.

Sources and further reading