Lesson objective
Identify the points in a brokerage operation where AI output stops being a draft and starts driving an action, and design the system so a wrong answer is caught before it becomes a wrong action.
Plain-language explanation
Why this matters now
At Level 2 an AI output was a draft. A licensee read it, corrected it and decided whether it went anywhere. In an operation, outputs increasingly go somewhere on their own: a field extracted from a contract becomes a calendar deadline, a lead assistant's answer is sent to a consumer, a set of listing details flows into an MLS input form, a summary becomes the note the next person relies on. The moment an output drives a step, a wrong answer is no longer a wrong draft. It is a wrong action.
Professional guidance (voluntary framework): the OWASP GenAI Security Project's Top 10 for LLM Applications 2026, dated August 4, 2026, is a community consensus list of the main risks in applications built on large language models. It is not a regulation and nobody can be certified to it. Its 2026 order is: LLM01 Prompt Injection; LLM02 Sensitive Information Disclosure; LLM03 Excessive Agency; LLM04 Supply Chain; LLM05 Data and Model Poisoning; LLM06 Unbounded Consumption; LLM07 Misinformation; LLM08 Hidden Context Exposure; LLM09 Vector and Embedding Weaknesses; LLM10 Improper Output Handling. It replaces the 2025 edition, whose numbering is no longer current.
The project leads explain one ranking in words every brokerage operation should keep in view. Voters placed Misinformation near the bottom, but the incident record placed it near the top: "When a model's fluent, confident output drives a decision or a tool call, a wrong answer turns into a wrong action." The list's method gave the community vote three-quarters of the weight and incident data one quarter, drawn from 7,714 incidents of which 6,639 had enough detail to classify.
A question to sit with
Your transaction software can read an executed contract and create the file's deadline tasks automatically. It has worked on every contract you have spot-checked. What happens, and who finds out, on the day it reads one date wrongly on a file nobody spot-checked?
What goes wrong
The draft becomes the record. A summary that was meant to help a person read a document becomes the thing everyone reads instead of the document. When the summary is wrong, every later step inherits the error with more confidence than the first.
Output flows straight into a system. OWASP's LLM10, Improper Output Handling, is about model output passed to another component without being treated as untrusted. In brokerage terms, an extracted date written into a calendar, a generated field written into a listing form or a drafted message placed in an outbound queue has skipped the point where a person could have stopped it.
The tool is allowed to do too much. OWASP's LLM03, Excessive Agency, which the project says climbed to third place in 2026, concerns systems that can act with more permission or autonomy than the task needs. An assistant that can send messages from an agent's account, change a listing's status or book showings has agency; the question is whether every one of those powers is needed.
Instructions arrive inside data. OWASP's LLM01 states that large language models "make no architectural distinction between 'instructions' and 'data'" and that there is "no clean equivalent to parameterized queries." A brokerage tool that reads consumer enquiries, uploaded documents or third-party remarks can meet text written to steer it. This lesson addresses only the defence: limit what the tool can do, so that being steered cannot cause harm.
The reasoning looks right. Professional guidance (voluntary framework): NIST's Generative AI Profile (NIST AI 600-1, July 2024) defines confabulation as "the production of confidently stated but erroneous or false content," says confabulations "are a natural result of the way generative models are designed," and notes that models "sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect." A well-explained deadline is not evidence of a correct deadline.
The practical answer
The project leads' advice is the design principle for this whole level: "Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks."
Find every place output drives a step. Walk the workflow and mark each point where an AI output is used without a person re-reading it against the source. The table shows common brokerage examples. It is an AI Coalition Network recommendation, not a legal requirement.
| Where output drives a step | Wrong answer becomes | Break point to add |
|---|---|---|
| Dates extracted from an executed contract create deadline tasks | A missed or early deadline | A coordinator confirms each extracted date against the signed contract before the task is live |
| A lead assistant replies to a consumer enquiry | A wrong or discriminatory statement sent under the brokerage's name | Replies limited to approved content; anything else goes to a licensee |
| Listing details generated from notes fill an MLS input form | A wrong field syndicated to portals | The listing agent approves the field set against the property file before submission |
| A summary of an inspection report becomes the note the agent works from | A missed defect in advice to a client | The note links to the report; the agent reads the report before advising |
| An AI draft sits in an outbound email queue | A message sent without review | No automatic send; a named licensee releases each message |
Give the tool only the powers the task needs. For each tool, list what it can read, write, send and change. Remove every power the task does not require. Best practice: least privilege is usually taught as a rule about data; at Level 3 it applies to actions too. A drafting tool that cannot send cannot send something wrong.
Treat every output as untrusted until checked. Output that goes into another system is checked the way you would check a stranger's data: against the source, by a person or a rule that does not rely on the model's own judgement.
Prefer reversible actions. A deadline task that a coordinator must confirm, a message that sits as a draft, a listing change held for approval: each can be undone before it matters. A sent message, a published listing and a wire instruction cannot. Module 3 turns reversibility into a tiering criterion.
Keep money and legal effect out of reach entirely. No AI tool in a brokerage operation should send, confirm or change wire instructions, sign or negotiate a contract, or give legal advice. These are not tasks to be done carefully by AI; they are outside what the tool is permitted to do.
Remember this
- Once AI output drives a step, a wrong answer becomes a wrong action.
- OWASP's 2026 list puts Excessive Agency third and names Improper Output Handling tenth; it is voluntary consensus, not law.
- Build the system so that when the model is wrong, nothing important breaks: limit powers, add break points, prefer reversible actions.
- Instructions can hide inside the data a tool reads; the defence is limiting what the tool can do.
- AI never sends wire instructions, signs or negotiates contracts, or gives legal advice.
Real estate example
Larkfield Realty Group is a fictional brokerage whose transaction software offers to read each executed purchase contract and create the file's deadline tasks. Operations manager Tobiah Quarrie switches the feature on for all 40 agents in February, and coordinators stop entering dates by hand. Over the first ten weeks the software builds deadline tasks for 150 files. In April an agent notices that the fictional file for 18 Orrin Court shows the inspection period ending two days after the date written on the signed contract. The buyer had initialled a handwritten change in the margin, and the software had read the printed date. Tobiah pulls every file where the contract carried a handwritten or initialled change, 22 in all, and has a coordinator compare each extracted date with the signed pages: 5 of the 22 carry at least one wrong date, while a sample of 30 files without handwritten changes shows none. Two of the five were caught in time only because an agent happened to re-read the contract. Tobiah does not switch the feature off. He changes where it sits in the workflow: extracted dates now arrive as proposed tasks, and a coordinator must confirm each one against the signed contract, with any handwritten change highlighted, before it becomes live. He removes the software's permission to send reminder emails to clients directly; reminders now go to the coordinator. He records the change, the version date and the reason, and tells the agents what changed and why. All figures are fictional.
Responsible-use warning
Do not let an AI output move into a calendar, an MLS form, an outbound queue or another person's working notes without a named checkpoint. The error that does harm is rarely the wrong draft a licensee reads; it is the wrong answer that nobody reads because a system acted on it. Do not give a tool powers the task does not need, and never let any AI tool send or confirm wire instructions, sign or negotiate a contract, or give legal advice.
Human-review requirement
The broker or delegated operations manager owns the list of places where output drives a step, the powers each tool holds and the break points on each arrow. The coordinator owns each extracted date confirmed against the signed contract. The licensee owns every message and listing field released under their name. Legal effect of contract terms stays with an attorney, and settlement figures and funds instructions stay with the closing or settlement agent. Nobody signs off an automated step they have not seen working against real documents.
Practical exercise
Use fictional material only. No real contract, client, property or agent.
- Choose one brokerage workflow that uses, or plans to use, an AI tool: contract-to-calendar, lead reply, MLS input or document summary. Write one sentence on who relies on its output today.
- Draw it as boxes and arrows. Mark each box where AI produces output.
- For every arrow leaving an AI box, write where the output goes and whether a person re-reads it against the source before it arrives.
- List every power the tool holds: read, write, send, change, delete. Strike out each power the task does not need.
- For each remaining arrow, write the wrong action a wrong answer would become, and whether it could be undone before it mattered.
- Add a named checkpoint to every arrow where a wrong action cannot be undone, and say what the checker compares the output with.
- Recompute this fictional result: 5 wrong dates in 22 contracts with handwritten changes, none in 30 without. State the rate for each group and what it suggests about where to check.
- Confirm in writing that the tool cannot send or change wire instructions, sign or negotiate anything, or give legal advice, and name the person who checked its settings and the date.
Knowledge check
-
What did the OWASP 2026 project leads say happens when fluent output drives a decision or a tool call?
Answer guidance: They said that when a model's fluent, confident output drives a decision or a tool call, a wrong answer turns into a wrong action, which is why Misinformation ranked near the top of the incident record.
-
Name two OWASP 2026 risks that apply when AI output flows into another brokerage system.
Answer guidance: LLM03 Excessive Agency, where a tool can act with more permission than the task needs, and LLM10 Improper Output Handling, where output passes to another component without being treated as untrusted.
-
What design principle did the OWASP project leads recommend?
Answer guidance: Stop trying to build a model that cannot be fooled, and build the system around it so that when the model is fooled, nothing important breaks: limited powers, break points and reversible actions.
-
Is the OWASP Top 10 for LLM Applications a legal requirement for brokerages?
Answer guidance: No. It is a voluntary community consensus document, not a regulation or a certifiable standard, though it is a useful map of where AI applications fail.
Sources and further reading
- OWASP Foundation, OWASP GenAI Security Project — OWASP Top 10 for LLM Applications 2026 (2026-08-04). Accessed 2026-09-15.
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- South Carolina General Assembly — South Carolina Code of Laws §40-57-820, Work product (2024-05-21). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.