Lesson objective
Capture a measured baseline for one workflow before redesigning it, and explain why published studies describe how gains were distributed but never supply a figure a small business should expect.
Plain-language explanation
Why this matters now
Every redesign in this level ends with the same question: did it help? That question can only be answered if the business wrote down how the work performed before anything changed. Once the new workflow is running, the old numbers cannot be recovered. People remember the old way as slower or faster than it was, depending on whether they liked the change.
The temptation is to borrow a number instead. Vendors quote research, and the research is real. But every figure this programme can cite is a laboratory task, a single large firm's rollout, a vendor-affiliated preprint or a self-report. None was measured in a small business, and none is a target. What those studies can teach is the shape of a result: who gained, on which tasks, and where the gains reversed. That shape is exactly why a business needs its own baseline, broken down by task and by person.
Research (single-firm rollout, not a randomised trial): Erik Brynjolfsson, Danielle Li and Lindsey Raymond, NBER Working Paper 31161 (issued April 2023, revised November 2023), studied 5,179 customer support agents at one software firm given access to a generative-AI conversational assistant. Access increased issues resolved per hour by 14% on average, but the gains were concentrated among less experienced and lower-skilled workers, about 34%, with minimal effect on the most experienced agents. The authors also report improved customer satisfaction and higher employee retention. This was a staggered rollout at a single firm, with scripted workflows and an assistant trained on that firm's own top-performer transcripts. NBER working papers are not peer reviewed, and the result concerns support-ticket resolution only.
Research (field experiment on consulting tasks): Fabrizio Dell'Acqua and colleagues, in Organization Science (vol. 37, issue 2, March 2026; earlier circulated as Harvard Business School Working Paper 24-013, 22 September 2023), randomly assigned 758 Boston Consulting Group consultants to no AI, GPT-4, or GPT-4 with a prompt engineering overview. On 18 tasks inside the AI capability frontier, AI users completed 12.2% more tasks and completed them 25.1% more quickly, with significantly improved quality. On a complex task selected to be outside the frontier, the published abstract says subjects using AI "were 19% less likely to produce correct solutions compared with those without AI." The 2023 working paper reported gains of 43% for below-average and 17% for above-average performers. These were consultants using GPT-4 as of 2023 on experimental tasks, not small-business staff.
A question to sit with
Six months from now, your office manager says the new AI-assisted quoting has "saved loads of time" and your newest helper says it has made more work. What numbers, written down this month, would let you tell which of them is right?
What goes wrong
Borrowing a target. A business adopts "14% faster" or "25% quicker" as its goal. Neither figure was measured on its work, its staff or its tools, and a result that falls short of a borrowed number is read as failure when it may be a success, or the reverse.
Measuring after the change. The baseline is collected once the new workflow is already running, so there is nothing to compare it against except memory.
One average hides the spread. Both studies found gains concentrated in some people or some tasks and absent, or reversed, in others. A single business-wide average can show a small improvement while the most experienced person's work gets worse and a new helper's gets better, or while one task speeds up and another starts producing wrong answers.
Counting only speed. Time to reply is easy to measure and flattering. A reply that is faster but wrong more often creates rework, complaints and refunds later. Dell'Acqua and colleagues' outside-the-frontier result is a reminder that the same tool can make a task quicker and less correct.
An unusual month as the baseline. A baseline taken in the week before Christmas, during a staff absence or straight after a price rise describes that week, not the business.
The practical answer
Map first, then measure. Professional guidance (voluntary framework): NIST's AI RMF 1.0 (January 2023) says that, after instituting its GOVERN outcomes, "most users of the AI RMF would start with the MAP function" and continue to MEASURE or MANAGE, iteratively. It is voluntary, binds no business and is being revised. The practical meaning: understand how the work actually runs, then count it. Module 2 builds the map; this lesson sets up the counting.
Choose measures that would show failure, not only success. A baseline needs at least one measure that would get worse if the change went badly.
| Measure | How to count it before the change | What it would reveal |
|---|---|---|
| Volume | Items handled per week, from the inbox or the job log | Whether the workload itself changed |
| Time to first reply | Received time to sent time, from timestamps | Speed, the measure most likely to flatter |
| Correction rate | Items changed by the checker before sending, out of items checked | Whether drafts are right, not just fast |
| Rework | Items sent that later had to be corrected or reissued | Errors that escaped the check |
| Complaints and refunds | Count and value, from the complaint log and the books | Harm that reached customers |
| Checker time | Minutes a day spent checking, logged for a sample of days | Whether the work moved rather than shrank |
Six measures, and at least four of them could reveal a problem.
Measure by person and by task where the business is big enough. Because gains in both studies were uneven across people and tasks, record the baseline separately for each task, and, where two or more people do the same task, for each person. Otherwise the average can hide a helper who is struggling and an experienced person whose work has slipped.
Take four normal weeks. Pick weeks without a holiday, a price change or a staff absence, and note anything unusual in the log. For seasonal businesses, note the season, and do not compare a summer baseline with a winter result.
Write the decision rule before the change. Decide now what result would mean continue, narrow or stop, for example: "If the correction rate rises above the baseline 7.5% for two consecutive weeks, the owner narrows the task." Module 14 builds the full measurement plan. Writing the rule first protects the decision from whoever likes or dislikes the change.
Remember this
- Capture the baseline before anything changes; afterwards, only memory is left.
- Published studies describe how gains were distributed, uneven across people and tasks, not what a small business should expect.
- Include measures that would show failure: correction rate, rework, complaints and checker time, not just speed.
- Record by task, and by person where the business is big enough, over four normal weeks.
- Write the continue, narrow or stop rule before the change goes live.
Small business example
Silverline Plumbing & Heating is a fictional six-person firm. The owner, Gwen Achterberg, plans to have an AI tool draft quote replies from enquiry forms, and the vendor's brochure promises replies "up to 25% faster, proven by research". Gwen reads the source and finds it was consultants on experimental tasks using GPT-4 in 2023. She decides the only number that matters is her own. For four normal weeks in February, avoiding the half-term week when one plumber was off, her office manager, Oluwaseun Price, logs every enquiry: 38, 42, 35 and 45, a total of 160, an average of 40 a week. Oluwaseun, who has worked there nine years, handled 100 of them; the newer office assistant, Jody Kerr, handled 60. Oluwaseun corrected 4 of his own drafts before sending (4%) and Jody corrected 8 of hers (about 13.3%), a combined 12 of 160, 7.5%. Three quotes had to be reissued after sending, all Jody's. Median time to first reply was 5 hours. Oluwaseun also logged his checking time on five days: 20, 25, 15, 30 and 20 minutes, a total of 110, an average of 22 minutes a day. Gwen writes the decision rule before switching the tool on: continue if the combined correction rate stays at or below 7.5% and reissues stay at or below 3 per 160 quotes; narrow the task if either measure is worse for two consecutive weeks; stop if a reissued quote costs the firm money. She also decides to read the results for Oluwaseun and Jody separately, because a combined average could hide a change in either direction for one of them. The baseline sheet is dated and filed with the procedure.
Responsible-use warning
Do not adopt a published figure such as 14% or 25.1% as the target for your business, and do not start the new workflow before the old one has been measured. Those figures come from one software firm's support agents and from consultants on experimental tasks, and neither is a small business. Do not rely on speed alone: a faster reply that is wrong more often costs more later. And do not take the baseline in an unusual month, or you will be comparing the new workflow with a week that never represented the business.
Human-review requirement
The owner approves the choice of measures, the baseline period and the decision rule before any change goes live, and signs off the dated baseline sheet. The operation owner collects the counts the same way before and after, and notes anything unusual. The owner reads results by task and by person, not only as an average, and makes the continue, narrow or stop decision against the rule written in advance. Any figure quoted to staff or customers about the change is the business's own measurement, stated with its period and method, never a borrowed research figure.
Practical exercise
Use a fictional business or anonymised counts with no customer names.
- Choose one workflow you plan to redesign with AI in the next three months.
- Pick four normal weeks for the baseline and write why each counts as normal.
- Choose at least four measures from this lesson's table, including at least two that would get worse if the change went badly.
- For each measure, write exactly where the count will come from and who will record it.
- If two or more people do the task, set up the baseline sheet so each person's counts are recorded separately.
- Record the baseline and calculate each rate, showing the division.
- Write the decision rule: the result that means continue, the result that means narrow, and the result that means stop.
- Write one sentence explaining why no published study figure appears on your sheet, and date and file the sheet with the procedure.
- List anything that could make the after-change weeks unlike the baseline weeks, such as a season, a price rise or a new starter, and write how you will note it.
- Decide how many weeks after the change you will measure before reading the results, and put the review date in the diary.
- Name who will read the results with you, so the person who championed the change is not the only one judging it.
Knowledge check
-
What did the Brynjolfsson, Li and Raymond study find about how gains were distributed, and what kind of study was it?
Answer guidance: Issues resolved per hour rose 14% on average, concentrated among less experienced and lower-skilled agents at about 34%, with minimal effect on the most experienced. It was a staggered rollout at one software firm, not a randomised trial, and is not peer reviewed.
-
Why can the Dell'Acqua experiment not supply a target for a small business?
Answer guidance: Its subjects were 758 management consultants using GPT-4 as of 2023 on experimental tasks. Its gains applied inside the capability frontier, and outside it AI users were 19% less likely to produce correct solutions, so it describes a shape, not a figure to expect.
-
Why should a baseline include measures other than speed?
Answer guidance: Speed is easy to count and flattering. A faster reply that is wrong more often creates rework, complaints and refunds, so correction rate, rework, complaints and checker time are needed to reveal failure.
-
A firm logged 160 enquiries in four weeks and corrected 12 drafts. What is the correction rate?
Answer guidance: 12 ÷ 160 = 0.075, a correction rate of 7.5%, which becomes the baseline figure the same measure is compared against after the change.
Sources and further reading
- National Bureau of Economic Research (Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond) — Generative AI at Work (NBER Working Paper 31161) (2023-04-01). Accessed 2026-09-15.
- Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani; Organization Science — Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (2026-03-01). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023-01-26). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.