Lesson objective

Apply the three-question test of consequence, reversibility and ownership to a proposed agent action, and record the decision before any authority is granted.

Plain-language explanation

Why this matters now

Lesson 2 gave the ladder. This lesson gives the test a plant applies before it moves any action up the ladder. It is short on purpose. A test that takes a day will be skipped; a test that fits on one page will be used every time.

The security community has converged on the same design stance.

  • Professional guidance (OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2026; voluntary community document): Excessive Agency is LLM03:2026, having climbed to third. The document maps it, in agentic systems, to three entries of the OWASP Top 10 for Agentic Applications for 2026: ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse and ASI08 Cascading Failures. The 2025 numbering is superseded and should not be quoted as current.
  • The project leads' letter in the 2026 list states the stance in two sentences: "Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks."
  • Professional guidance (CISA and partners, May 2026; voluntary): CISA recommends that organisations "account for agentic AI security in your organization's security model and risk posture."
  • Professional guidance (NIST, AI RMF 1.0, January 2023; voluntary): NIST states that after instituting GOVERN outcomes, "most users of the AI RMF would start with the MAP function" and continue to MEASURE or MANAGE, iteratively. Asking these questions before connecting anything is a practical form of mapping.

A question to sit with

For the agent your team is most excited about, can you name the person who would answer for its worst action, and have you asked them?

What goes wrong

The test is about accuracy instead of consequence. The team asks "how often is it right?" and never asks "what happens on the run where it is wrong?" An agent can be right on almost every run and still do serious harm on the one where it is not.

Reversibility is judged from inside one system. The record can be edited back, so the step is called reversible. But the planning run, the customer portal or the shift lead already acted on it.

The owner is the enthusiast. The person who wanted the agent approves it. The person who owns the result it affects, such as the quality manager, the maintenance manager or the materials manager, hears about it later.

The test is run once. The answers were right for the first version. Then the agent gained a new tool, a new system or a new product line, and nobody asked the questions again.

The model is assumed not to be fooled. Supplier documents, scanned paperwork and inbound email can contain text that reads like an instruction. The test assumed the agent would always follow the plant's instructions and not the document's. Module 10 treats this from the defender's side.

The practical answer

Ask three questions, in writing, for every action. The test below is an AI Coalition Network recommendation. It is not taken from OWASP, NIST or CISA, and it does not paraphrase any OWASP entry.

  1. Consequence and reach. If the agent were wrong, or had been fooled by something it read, what is the worst this action could do, and what would it reach: which records, which systems, which people, which customers or suppliers, which parts?
  2. Reversibility. Can the plant restore the prior state completely, from its own records, before anyone on the floor, any customer, any supplier or any other system acts on the change?
  3. Ownership. Who is the named person who owns the result, and do they see the action before it takes effect?

Turn the answers into a ceiling. The decision table below converts the three answers into the highest rung the action may run on.

Consequence and reach Reversible by the plant alone? Owner sees it first? Highest rung
Could command equipment Any Any None: out of scope for this tier
Leaves the plant or changes a part's status No Yes, releases each one Rung 3
Leaves the plant or changes a part's status No No Refuse as designed
Stays inside office systems, limited reach Yes After the fact, in review Rung 2
Stays inside office systems, wide reach Yes After the fact, in review Rung 1 until reach is narrowed
Stays inside office systems No Yes Rung 1 or rung 3

The table has six rows; read down until the first row matches. "Wide reach" means more than one product line, customer programme or site, or more records than a person could check in a normal review.

Work the test on real proposals. Here are five fictional proposals from a mid-sized plant, tested the same way.

Proposal Consequence and reach Reversible? Owner sees first? Decision
Summarise shift handover notes into a morning brief A misleading brief; one reader Yes, nothing is changed The production manager reads it Rung 1
Create draft CMMS work orders from downtime notes A wrong draft against one asset Yes, if drafts are internal and unassigned The planner reviews drafts daily Rung 2 for drafts only
Change lead times in the ERP item master from supplier emails Planning runs order the wrong quantities across many parts No, once planning has run No Refuse; draft the change for the planner at rung 1
Send requests for quotation to approved suppliers A wrong drawing revision or quantity reaches a supplier No, the email has left The buyer releases each one Rung 3
Adjust a tool-wear offset on a CNC from inspection data Commands a machine Not relevant Not relevant Out of scope for this tier

Five proposals, five decisions, and only one runs on its own inside the office systems.

Assume the agent will be fooled. Following the OWASP leads' stance, ask question 1 twice: once for an honest mistake, and once for the case where a document the agent read told it to do something else. If the worst case in both is "a wrong internal draft a person will catch", the design is sound. If it is "a wrong message to a customer" or "a wrong status on a lot", the design is not.

Name the owner by role and by person. "Quality" is not an owner. "The quality manager, currently R. Osei, deputy the senior quality engineer" is. The owner is the person who would answer for the result to a customer, an auditor or a regulator. They sign the test before anything is connected.

Re-run the test on every change. New tool, new system, new data class, new product line, new customer programme: each is a new answer to question 1. Record the test with the agent's register entry, dated, and treat any change as a reason to repeat it.

Why "it is usually right" is not a design argument

An agent runs many times a week. A tool that is right almost every time will still be wrong on some run, and the design decides what that wrong run can touch. Accuracy lowers the frequency of bad outcomes; scope, reversibility and ownership limit their size. Only the second set is fully in the plant's control.

Remember this

  • Ask three questions of every action: consequence and reach, reversibility by the plant alone, and whether the named owner sees it first.
  • Convert the answers into a ceiling with the decision table, and refuse any action that fails all three.
  • Ask question 1 twice: once for an honest mistake and once for an agent that has been fooled.
  • The owner is a named person who would answer for the result, and they sign before connection.
  • OWASP maps Excessive Agency (LLM03:2026) to ASI02, ASI03 and ASI08; the lists are voluntary, and the test is this programme's own.

Manufacturing example

Hollow Creek Stamping is a fictional 90-person stamping and assembly plant supplying brackets and housings to automotive and appliance customers. Its materials manager, Luis Ferrante, proposes an agent that reads supplier emails, updates lead times and minimum order quantities in the ERP item master, and then sends a revised schedule to each affected supplier. The manufacturing IT lead, Hana Okoro, runs the three-question test with him. On consequence and reach, the item master drives the overnight planning run for all 1,850 active purchased parts, so one misread email could change planned orders across every customer programme, and a revised schedule would reach suppliers. On reversibility, the field can be edited back, but by morning the planning run will have generated suggested orders from the wrong value and the supplier will have read the schedule. On ownership, Luis owns the item master, but nobody would see a change before the planning run used it. The action fails all three questions, which the decision table says to refuse. Hana then asks question 1 a second time, assuming a supplier email contained text written to make the agent change a value; the worst case is the same, which confirms the refusal. The redesign splits the proposal into three actions. The agent reads the emails and drafts proposed item master changes in a review list, which is rung 1. Luis or a planner applies approved changes before the planning run, as a person's act. The revised schedule is drafted by the agent and released by the buyer, one supplier at a time, at rung 3. Each action gets its own one-page test, signed by Luis as owner, and the register records the three rungs.

Responsible-use warning

Do not run the three-question test from inside the one system the agent touches. An ERP field can be edited back, but the planning run, the supplier, the customer portal or the shift lead may already have acted on it. Test reversibility against everything downstream, and test consequence twice: once for an honest mistake and once for an agent that was fooled by something it read. An action that fails all three questions is refused, not accepted with extra monitoring.

Human-review requirement

The owner of the result signs the test: the quality manager or the person the quality system names for quality records, the materials manager for purchasing and planning data, the maintenance manager for CMMS actions, and the relevant customer programme manager where a customer's requirements decide who may approve. They check that reach was traced through downstream systems, that reversibility was tested against what others can see, and that the test is repeated whenever the agent gains a tool, a system or a product line.

Practical exercise

Use a fictional plant, or your own with nothing confidential.

  1. Choose one agent proposal your plant has discussed, and break it into its separate actions, one sentence each.
  2. For each action, answer the consequence and reach question for an honest mistake, naming every record, system, person, customer, supplier or part it could reach.
  3. Answer the same question again, assuming a document the agent read contained text written to make it do something else, and note whether the worst case changes.
  4. For each action, trace every downstream system that could act on the change before the plant reverses it, and answer the reversibility question.
  5. Name the owner of each action by role and by person, and state whether they would see the action before it takes effect.
  6. Use the six-row decision table to set the highest rung for each action, recording which row matched first.
  7. For any action the table refuses, redesign it into smaller actions that pass, and test each again.
  8. Write the result on one page per action, with the date and a signature line for the owner.
  9. List three changes that would require the test to be repeated for this agent, and state who would notice each change.

Knowledge check

  1. What are the three questions in this lesson's test?

    Answer guidance: Consequence and reach if the agent is wrong or fooled; whether the plant alone can fully reverse the action before anyone or anything acts on it; and who the named owner is and whether they see the action before it takes effect.

  2. Why is the consequence question asked twice?

    Answer guidance: Once for an honest mistake and once for an agent fooled by content it read, following the OWASP leads' stance that the system should be built so nothing important breaks when the model is fooled.

  3. Which combination of answers does this lesson say to refuse outright?

    Answer guidance: An action that reaches outside the plant or changes a part's status, cannot be fully undone by the plant alone, and takes effect before a named owner sees it.

  4. How does the OWASP LLM Top 10 2026 map Excessive Agency in agentic systems?

    Answer guidance: Excessive Agency is LLM03:2026 and is mapped to ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse and ASI08 Cascading Failures. Both OWASP lists are voluntary community documents.

Sources and further reading