Open almost any AI policy and you will find some version of the sentence: a human reviews the output before a decision is made.
It is the most common control in AI governance and the one least likely to be true. Not because anyone is lying — the reviewer exists, the step is in the workflow, the box gets ticked. But the conditions that would make the review meaningful are usually absent, and nobody checked whether they were present, because the sentence sounded sufficient.
This matters more as the control does more work. It is the answer given when a system is high-risk. It is what makes a deployment defensible to a board, a regulator or a resident. Regulatory frameworks lean on it heavily. If it is nominal, a great deal is resting on nothing.
Why review fails, predictably
Automation bias
People over-trust automated recommendations. This is one of the better-established findings in human-factors research, and it has an uncomfortable property: the effect gets stronger as the system gets better.
A system that is wrong half the time keeps a reviewer alert. One that is right 95% of the time trains them, over weeks, that checking is wasted effort. And the 5% is exactly where oversight was supposed to earn its place.
So reliability does not reduce the need for oversight design. It increases it, which is the opposite of most people's intuition and of most procurement conversations.
The reviewer cannot see the reasoning
Much review is arranged so the person sees a conclusion — a score, a flag, a recommendation — and not the material behind it. That person cannot evaluate the reasoning. They can only decide whether they feel like disagreeing with a number, which is not the same activity.
This is where the limits of explainability bite in practice. If the system cannot show its working, and the reviewer has nothing but its output, "review" collapses into "acceptance with extra steps."
Throughput expectations moved
Here is the pattern that quietly kills more oversight than any technical factor.
A tool is introduced to make a process faster. Case volumes per person rise, because that was the point. The review step is written into the policy. Nobody recalculates how long meaningful review takes, or notices that the time available per case just fell by two-thirds.
If throughput expectations rose when the tool arrived, review was designed out at the moment it was written down. The reviewer is not being careless; they are doing the only thing the arrangement permits.
Disagreeing is expensive
If overriding the system means writing a justification, triggering a second review, or explaining yourself to a manager who is measured on throughput, then agreement is the cheap path and disagreement is the expensive one. People take the cheap path, and they are right to, because that is what the organisation is asking for whatever the policy says.
The four conditions
Oversight is real when four things hold. They are structural — none is fixed by telling reviewers to be more careful.
1. The reviewer sees what the system saw
Not just the output. The inputs, the relevant record, the material that drove the conclusion. A reviewer needs enough to form an independent view, otherwise they are auditing a number rather than a decision.
Practical test: could this reviewer reach a different conclusion using what is in front of them? If not, they are confirming, not reviewing.
2. There is time
Work out how long an actual review takes on a real case. Compare it to time available per case. If the second number is smaller, you do not have review — you have a queue with a signature at the end.
This is arithmetic, and it is worth doing before an incident rather than after.
3. Disagreement is normal and cheap
Make overriding the system as easy as accepting it. Then watch the rate.
An override rate of zero is a finding, not a reassurance. It means either the system is perfect — it is not — or the reviewers have stopped. Something in the range of a few percent usually indicates people are still looking. Sustained zero means the control has lapsed and nobody noticed because nothing broke visibly.
Track it. It is the cheapest health check available for the most-claimed control in your programme.
4. Accountability sits on the outcome, not the queue
If the reviewer is measured on cases cleared, they will clear cases. If they are accountable for whether the decisions were right, they will look.
Both cannot be primary. Whichever is measured is the one that actually governs, regardless of what the job description says.
When human review is the wrong control
Sometimes the honest answer is that review will not work, and pretending otherwise is worse than admitting it. Three cases:
- The volume makes it impossible. If a system processes ten thousand items a day and you have two reviewers, no arrangement makes that meaningful review. Consider sampling with a real audit, or narrowing what the system decides autonomously.
- The reviewer cannot have the expertise. Review of a specialist determination by a non-specialist is theatre. Either resource it properly or accept that this is an automated decision and govern it as one.
- The output is unfalsifiable to the reviewer. If there is no way for the person to tell a good output from a plausible bad one — which is common with generated text — review catches format problems and misses substance.
In each case the better move is to change what the system is permitted to decide, rather than to keep a control that exists only on paper. That is a governance decision, and it belongs to whoever owns AI risk. See AI governance.
Where agents make this harder
Everything above assumes a system that recommends and a person who decides. Agents invert it: the system acts, and the human is asked to approve or intervene.
That is a harder oversight problem, because the failure of a confirmation dialog is silent. A prompt saying "the agent would like to proceed" trains people to click yes within a day. To be meaningful, a confirmation must state the specific action and its specific target, in terms the person can evaluate without reconstructing the agent's reasoning.
Joint guidance from CISA, the NSA, Australia's ACSC and international partners in April 2026 recommends human sign-off for high-impact agent actions. That recommendation only means something if the sign-off is designed to the conditions above; otherwise it is the same rubber stamp at higher speed. See prompt injection for why the surrounding controls matter as much as the approval step.
A diagnostic you can run this week
Pick one system where your policy claims human review. Ask:
- What exactly does the reviewer see? Get a screenshot rather than a description.
- How long do they have per case? Divide, do not estimate.
- What is the override rate over the last quarter? If nobody knows, that is the answer.
- What happens to a reviewer who overrides — any extra work, any explaining?
- Are they measured on cases cleared or on decisions being right?
- When did anyone last check that this control works?
If the answers are uncomfortable, that is useful. A control you know to be nominal can be fixed, narrowed or replaced. A control you believe in and is nominal is the one that fails in front of somebody who has been harmed by it.