Lesson objective
Explain from the field experiment why the same person can do well with AI on one task and badly on the next, and design a session around a checking habit that transfers across tools and tasks.
Plain-language explanation
Why this matters now
Owners often ask an instructor which tool to use. It is a reasonable question and the wrong centre for a session. Tools change their features, their prices and their terms, sometimes within months. A session built around one tool's buttons is out of date by the time the owner needs it, and it teaches nothing about the moment the tool is wrong.
A checking habit lasts longer. The best evidence this programme holds for why a habit matters more than a tool comes from an experiment on consultants, not on small business owners, and it has to be taught with its limits.
Research (not binding): Dell'Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and Lakhani, "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality", Organization Science, vol. 37, issue 2 (March 2026), earlier circulated as Harvard Business School Working Paper 24-013 (22 September 2023). It was a preregistered experiment with 758 consultants at Boston Consulting Group, randomly assigned to no AI, GPT-4, or GPT-4 with a prompt engineering overview. The AI was GPT-4 as of 2023.
- For 18 tasks inside what the authors call the AI capability frontier, AI users completed 12.2% more tasks, completed them 25.1% more quickly, and produced significantly higher quality work.
- For a complex task selected to be outside the frontier, the published abstract reports that subjects using AI "were 19% less likely to produce correct solutions compared with those without AI." The 2023 working paper put it as "19 percentage points less likely"; quote the published wording and say which version you cite.
The same people, with the same tool, did better on some tasks and worse on another. That is the finding you may teach: performance with AI was uneven across tasks. It was measured on management consultants doing a consulting task set, with a 2023 model. It is not a measure of what owners will gain, and it is not a learning outcome.
A question to sit with
An owner in your session says, "It writes my social posts brilliantly, so I've started letting it answer customer questions about our returns policy too." What would you want them to understand about why the second task is different from the first?
What goes wrong
1. The tool tour. The session teaches one product's menus. The owner changes tools, or the product changes, and nothing learned applies.
2. Borrowing the gain. The instructor tells owners they can expect to work 25% faster. That figure described consultants on a set of consulting tasks with a 2023 model, and it may not be borrowed for a bakery.
3. Trust spreading from the good task to the bad one. Good results on one kind of task become confidence on a different kind, where the tool fails.
4. The check that depends on the tool. An owner learns to "ask the AI whether it is sure". NIST AI 600-1 notes that large language models "sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect." The tool's confidence is not a check.
5. The persuasive wrong answer. NIST AI 600-1 describes automation bias as "excessive deference to automated systems" and says it "can exacerbate other risks of GAI, such as risks of confabulation." A fluent, well-explained answer invites exactly that deference.
The practical answer
Best practice: teach the habit, then let the tool vary. The checking habit in this programme has four moves, and none of them depends on a particular product.
| Move | What the owner does | Works with any tool because... |
|---|---|---|
| Notice | Recognise that a figure, rule or promise came from the tool | It is about the output, not the menus |
| Source | Open the business's own source: rate sheet, policy, supplier sheet, schedule | The source belongs to the business |
| Compare | Check each figure, rule and promise against that source | It needs no feature of the tool |
| Record | Note that the check was done and by whom | It lives on the owner's paper or file |
Best practice: teach owners to sort their own tasks. The experiment's frontier cannot be looked up for a business, and nobody publishes one for plumbers or florists. What an owner can do is sort their own tasks by what happens if the tool is wrong.
| The tool is wrong and... | Examples | What the habit requires |
|---|---|---|
| Nobody outside sees it | Brainstorming names, summarising the owner's own notes | Light check before use |
| A customer sees it | Quotes, replies about policy, product descriptions | Full check against source before sending |
| Money, law, health or a licence is involved | Refunds, contracts, allergen claims, tax treatment | The named person decides; the relevant professional is consulted where the work is theirs |
This sorting is a practice, not the experiment's frontier and not a finding. It is how an owner turns "performance is uneven" into a decision about where to spend the check.
Best practice: use the framework's language sparingly. NIST AI RMF 1.0 notes in Appendix C that human-AI configurations can span "from fully autonomous to fully manual," and that in some conditions the AI part of a human-AI interaction "can amplify human biases, leading to more biased decisions than the AI or human alone." The framework is voluntary and under revision, and it binds no small business. The owner-sized version is the sorting table above: decide, task by task, how much the tool may do before a person checks.
Keep claims to customers inside the habit. When the output is a claim to a customer, remember that FTC Act Section 5 declares "unfair or deceptive acts or practices in or affecting commerce" unlawful, with no size threshold and no AI-specific provision. The checking habit is how an owner avoids sending a claim the tool made up. Whether a particular claim is lawful is a question for that business's attorney.
Remember this
- Teach a checking habit that works with any tool: notice, source, compare, record.
- The field experiment found uneven performance across tasks for consultants using a 2023 model; teach that, never a gain to expect.
- Quote "19% less likely" from the published version and say which version you cite.
- Owners sort their own tasks by what happens if the tool is wrong; that sorting is practice, not a finding.
- A fluent explanation from the tool is not a check.
Small business example
Nadia Ferreira, a fictional small business adviser for the fictional Riverside Enterprise Hub, is asked by an owner group to "teach us the best AI tool". She declines to rank products and builds a session around the checking habit instead. Her room includes Tomasz Wilk, a fictional owner of a four-person bicycle repair shop, who says the tool "never gets anything wrong" because his social posts have all been good. Nadia gives every owner two fictional tasks for a fictional bicycle shop. Task one: reword a friendly post about extended summer opening hours, with the hours supplied. Task two: answer a customer's question about whether a second-hand bike can be returned after thirty days, when the tool has been given no returns policy. On task one, every owner gets usable text. On task two, the tool writes a confident answer offering a sixty-day return window, explained with a plausible reason, although the fictional shop's written policy, printed on the back of the sheet, allows fourteen days for used bikes. Nine of the twelve owners would have sent it before turning the sheet over. Tomasz is one of them. Nadia then describes the Dell'Acqua experiment: consultants, a 2023 model, better results on most tasks and worse on one chosen to be outside what the tool could do. She quotes the published "19% less likely" wording and says it does not tell anyone how an owner will do. Each owner then sorts five of their own tasks into the three rows of the sorting table and writes the four moves of the habit on a card. Tomasz moves customer questions about returns into the "customer sees it" row and adds his written policy as the source.
Responsible-use warning
Never tell owners they can expect to work 25.1% faster, complete 12.2% more tasks or be 19% worse on hard problems; those figures describe consultants on a consulting task set with a 2023 model. Never present the experiment as a learning outcome, as evidence about small businesses, or as a frontier an owner can look up. Never rank or recommend a tool, and never teach "ask the AI if it is sure" as a check.
Human-review requirement
The instructor owns the account of the research and the fictional tasks. A second reviewer confirms that the Dell'Acqua experiment is described with its population, model, year and version, that the "19% less likely" wording is quoted from the published article, that no figure is presented as a gain owners should expect, and that NIST publications are described as voluntary. Each owner owns the sorting of their own tasks and the checks they then perform.
Practical exercise
Build a two-task demonstration of uneven performance. Use a fictional business and fictional documents throughout.
- Choose a fictional business in a trade likely to be in your room, and write a one-paragraph description of it.
- Write a fictional source document for it, such as a returns policy, a rate sheet or an opening-hours notice.
- Design task one, which the tool is likely to handle well because all the information is supplied.
- Design task two, which asks the tool about something it has not been given, where a confident wrong answer is likely, and put the correct answer in the source document.
- Write the three sentences you will say about the Dell'Acqua experiment, naming its population, model and year, quoting the published wording, and stating what it does not show.
- Write a sorting table of five tasks for your fictional business, placing each in one of the three rows and naming the source for each check.
- Write the four moves of the checking habit on a card, in words an owner with no technical background would use.
- Ask a colleague to review your script and mark any sentence that turns the experiment into a promise.
- Rewrite every marked sentence so it states the population, the model year and what was not measured.
Knowledge check
-
What may an instructor teach from the Dell'Acqua field experiment?
Answer guidance: That performance with AI was uneven across tasks: consultants using a 2023 model did better on tasks inside the frontier and worse on one chosen to be outside it; it is not a gain owners should expect.
-
Which wording should be quoted for the task outside the frontier?
Answer guidance: The published Organization Science abstract: subjects using AI "were 19% less likely to produce correct solutions", with the citation saying which version is being quoted.
-
What are the four moves of the checking habit?
Answer guidance: Notice that something came from the tool, open the business's own source, compare each figure, rule and promise against it, and record who checked it.
-
Why is asking the tool whether it is sure not a check?
Answer guidance: Because NIST AI 600-1 notes that models sometimes give logical steps even when the answer is wrong, so a confident explanation from the tool does not show the answer is right.
Sources and further reading
- Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani; Organization Science — Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (2026-03-01). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023-01-26). Accessed 2026-09-15.
- United States Congress (codified in the United States Code; enforced by the Federal Trade Commission) — 15 U.S.C. § 45 — Unfair methods of competition unlawful; prevention by Commission. Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.