Lesson objective
Explain why a tool's good results on one task say little about the next, and keep a per-task map of where the business's own tool has earned a lighter check and where it has not.
Plain-language explanation
Why this matters now
Most owners judge a tool by its best moment. It rewrote a clumsy notice beautifully, so it must be good. It summarised a long supplier contract in seconds, so it must be reliable. The next week the same tool is asked to apply a 15% discount across a mixed order, or to say whether a customer is owed a refund, and the owner brings the same trust.
That is the mistake this lesson is about. Trust earned on one task does not carry to the next.
For a small business, the useful point is the shape of the result, not its numbers. The same people with the same tool did better on some tasks and worse on another, and the tasks did not announce which kind they were. Nothing in the study measures a bakery, a garage or a salon.
A question to sit with
A fictional salon owner has used an assistant for six months to write appointment reminders and social captions, with no problems. Today she asks it to work out how much each stylist is owed from a commission sheet. What, if anything, has those six months told her about today's task?
What goes wrong
Trust transfer. Good results on writing tasks are treated as evidence for arithmetic, rules or judgement tasks.
The streak. Thirty clean reminders make the thirty-first feel checked. They are evidence about the thirty, not the next one, and not about a different task.
Busy-day checking. Checks are done last, in a rush, while the phone rings. Research on automated aids finds attention is the first thing lost.
The confident explanation. The tool explains why its commission figure is right, and the explanation is read as a check.
Mapping the tool instead of the task. "The assistant is fine for customer stuff" treats a birthday greeting and a refund decision as one category.
Believing training alone fixes it. A reminder to "always check carefully" is given once, and the business assumes the problem is solved.
The practical answer
Keep a task map, not a tool rating. List the tasks the business gives to the tool, and record for each one what the evidence from your own use actually shows.
| Task | Kind | Your own recorded results | Check level now |
|---|---|---|---|
| Appointment reminder from the booking list | Rewording supplied facts | Many recorded, date and time checked each time | Standard: date, time and name against the booking |
| Social caption for a supplied photo and offer | Rewording, but public | Several recorded; one invented "best in town" removed | Owner: every claim against the offer sheet |
| Commission calculation from a sheet | Arithmetic over many rows | None | Not a tool task until tested; recompute every line if used |
| "Is this customer owed a refund?" | A question of policy and possibly law | Not applicable | Route D for any legal part; policy answer from the written policy only |
Signs that a task may sit outside the frontier for your work. None of these is proof. Each is a reason to check harder or not to use the tool:
- The answer needs a fact you did not supply.
- The answer needs a rule: a law, a tax treatment, a licence condition.
- The task involves several calculation steps, percentages or many rows.
- The task needs a judgement about a person: a customer's honesty, a worker's performance.
- The task is new to your business, or is an old task with new inputs.
- The output would be hard to undo.
The same tool, a different job. The phrase that worked for captions will not make commission sheets safe. A task that is new gets a new triage, even if the tool, the account and the person are the same.
Why checking gets harder on a busy day. Research (a review of studies on automated decision aids, before today's language models; Human Factors, vol. 52, issue 3, June 2010): Parasuraman and Manzey report that "automation complacency occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention." They found complacency "in both naive and expert participants" and that it "cannot be overcome with simple practice." Automation bias "cannot be prevented by training or instructions" alone, and "can affect decision making in individuals as well as in teams." The review is about aids in fields like aviation, so it applies to a small business only by analogy. It does not say training is worthless; it says instructions alone did not prevent the bias.
What NIST says about the same risk. Professional guidance (NIST AI 600-1, July 2024; voluntary): the Generative AI Profile lists human-AI configuration among its twelve risks, including people "experiencing algorithmic aversion, automation bias, over-reliance, or emotional entanglement" with these systems. Section 2.7 calls automation bias "excessive deference to automated systems" and says it "can exacerbate other risks of GAI, such as risks of confabulation."
Design the check so it survives a busy day. Because instructions alone are not enough, build the check into the work.
| Control | How it works in a small business |
|---|---|
| Check at a planned time | Customer-facing drafts are checked at a set quiet slot, not squeezed in while serving |
| Check against the source, not the draft | The price list or policy is open beside the draft every time |
| Named comparisons | The check card lists three to five specific comparisons, not "review carefully" |
| A second pair of eyes for public output | Where there is more than one person, the one who drafted it is not the only one who reads a public post |
| Spot-check the streak | Once a week, redo one "routine" output completely by hand and compare |
| Distrust the explanation | A tool's reasoning about its own answer is not evidence; only the source is |
When a task earns a lighter check. Only your own record can say so, and only for that task with those inputs. Even then, Route B tasks keep the owner's check. A lighter check means fewer comparisons, never none.
Remember this
- Trust earned on one task does not carry to the next. Map tasks, not tools.
- Warning signs: unsupplied facts, rules, multi-step arithmetic, judgements about people, new inputs, hard-to-undo outputs.
- Complacency grows under busy conditions, in experts as well as beginners, and instructions alone do not stop automation bias.
- Build the check into the work: a planned time, the source open, named comparisons, a weekly spot-check.
- A tool's explanation of its own answer is not a check.
Small business example
Copperleaf Studio is a fictional three-chair salon owned by Hana Whitlock. For six months she has used an assistant to turn her booking list into appointment reminders, and her use record shows each one checked against the booking. In September she wants to use it for the monthly commission sheet: each stylist earns 40% of the service revenue they bring in. She triages it as a new task, even though the tool is the same. Her map marks it "arithmetic over many rows, no recorded results". She tests it on last month's sheet, which her bookkeeper had already checked. Stylist Ros had service revenue of $4,250, so 40% is $1,700. Stylist Kemal had $3,875, so 40% is $1,550. The tool returns $1,700 for Ros and $1,500 for Kemal, with a neat explanation of its method. Hana's own calculation shows Kemal's figure is $50 short. The explanation looked correct and was not. She records the result on her map: "commission sheet, tested once, one error in two lines; not a tool task". She keeps the calculation in her spreadsheet, where the formula gives the same answer every time, and uses the assistant only to word the note that goes with each payslip, checking the figure in the note against the spreadsheet. Her reminder task keeps its standard check. Once a week she redoes one reminder entirely by hand as a spot-check. In the third week the spot-check finds a reminder that gave a Thursday appointment as Tuesday, because the booking list had been pasted with a column out of place. The streak had not made that reminder safe, and she adds "day of the week" to the named comparisons on the reminder check card.
Responsible-use warning
Do not let a good month on one task lower the check on another. The strongest trust usually comes from easy rewording tasks, and it leaks into arithmetic, rules and judgements about people, where the tool may be weaker and a mistake costs more. A clear explanation from the tool is not a check. A reminder to "be careful" is not a check either: build the comparison into the work, with the source open, at a time when you can concentrate.
Human-review requirement
The owner keeps the task map and decides every change to a check level. The owner checks that each task on the map is described by its inputs as well as its kind, that any lighter check rests on recorded results for that exact task, that new tasks and old tasks with new inputs were triaged again, and that the weekly spot-check happened. Where a task depends on a rule, a tax treatment or a judgement about an employee, the owner confirms it is routed to an attorney, the signing preparer or the relevant licensed professional, and not to the map.
Practical exercise
Use a fictional business and fictional figures only.
- List six tasks your fictional business could give to an assistant, mixing rewording, arithmetic, a rule question and a judgement about a person.
- For each, write its kind and mark any of the six warning signs from this lesson.
- Create a fictional ten-row sales sheet with a percentage calculation, such as a commission or a discount, and compute every row yourself.
- Write a fictional tool output for the same sheet containing one wrong row and a confident explanation. Check it against your own figures and record what you found.
- Fill in a task map with four columns: task, kind, your recorded results, check level now.
- For each task, write the named comparisons you would put on its check card.
- Choose a planned checking time for your business and write why it is quieter than the time the drafts are made.
- Write your weekly spot-check: which output, redone how, compared with what.
- If your fictional business has one person, write how you will separate drafting from checking in time.
- Mark any task on your map that depends on a rule, a tax treatment or a judgement about an employee, and write who receives that question instead.
- Write the one recorded result that would have to exist before any task on your map could earn a lighter check, and what that lighter check would still include.
- Record the map and the spot-check plan in your workbook.
Knowledge check
-
Why does six months of good results on reminders say little about a commission calculation?
Answer guidance: They are different kinds of task. A tool can do well on one and badly on another, so trust earned on rewording does not transfer to arithmetic.
-
Under what condition did Parasuraman and Manzey find automation complacency occurs?
Answer guidance: Under multiple-task load, when manual tasks compete with the automated task for the operator's attention, in both naive and expert participants.
-
Name three of the six warning signs that a task may sit outside the frontier for your work.
Answer guidance: Any three of: a fact you did not supply, a rule, multi-step arithmetic, a judgement about a person, a new task or new inputs, an output hard to undo.
-
How can a one-person business separate drafting from checking?
Answer guidance: By separating them in time: draft at one point, then check later against the source with the check card in front of them.
Sources and further reading
- Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani; Organization Science — Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (2026-03-01). Accessed 2026-09-15.
- Raja Parasuraman and Dietrich H. Manzey; Human Factors (Human Factors and Ergonomics Society) — Complacency and Bias in Human Use of Automation: An Attentional Integration (2010-06-01). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.
- Complacency and Bias in Human Use of Automation: An Attentional Integration, Raja Parasuraman and Dietrich H. Manzey, Human Factors, 2010