Lesson objective

Identify whether a practice AI feature uses retrieval, tools or agent behaviour, and name the new failure and the first control each one introduces.

Plain-language explanation

Why this matters now

A model on its own only writes text. Many products sold to practices do more. They search a set of documents before answering, look things up in the practice management system, read an inbox of faxes or portal messages, or carry out several steps in a row. Each of those additions changes what can go wrong and who can influence the result.

The National Institute of Standards and Technology described one of the most important changes in Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025, published March 2025). It explains that generative AI models "combine the data and instruction channels," so someone can manipulate the resources a system reads, such as the material an internet-connected agent or a retrieval-augmented generation (RAG) system takes in. NIST calls this indirect prompt injection. It is mounted "not by the primary user of a model but instead by a third party," and often the primary user is the one harmed.

A practice receives a great deal of material written by outsiders: referral packets, faxed records, payer letters, portal messages and emails. Once an AI feature reads that material, the people who wrote it can affect what the feature does. NIST AI 100-2 is a voluntary taxonomy and terminology document, not a control standard, and it does not tell a practice what controls are required. It does explain why the design of these features matters.

A question to sit with

An assistant reads incoming referral faxes and can add tasks to the scheduling work queue. Before it was connected, only your staff could add those tasks. Who else can now put words in front of it?

What goes wrong

Assuming retrieval means the answer came from the document. A retrieval feature finds passages and gives them to the model, but the model still writes the answer. It can mix the passage with its own predictions, or cite a passage that does not say what the answer claims.

Retrieving the wrong or outdated document. A search over a folder of payer policies returns whatever matches best, including last year's version if it is still in the folder.

Giving a tool more reach than the task needs. A feature that only needs to read the schedule is given permission to book, cancel and message patients, because that was the default setting.

Forgetting that the inbox is written by strangers. Content from outside the practice can contain wording aimed at the AI rather than at a person. A feature that obeys it can take an action nobody in the practice asked for.

Not knowing what is inside the product. One product may combine a model from one company, a speech service from another and hosting from a third. NIST's Generative AI Profile names "Value Chain and Component Integration" among its twelve generative AI risks for this reason.

The practical answer

Learn four building blocks. Most products are a combination of them.

Building block What it adds New failure it brings First check or control
Model alone Writes text from its training and what you type Confabulation; out-of-date recall Verify every fact against a primary source
Retrieval Searches a document set and gives passages to the model Wrong, stale or partial passages; answers that go beyond the passage Open the cited passage and confirm it says what the answer claims; know who keeps the document set current
Tools Lets the model call other systems, such as looking up an appointment or an eligibility response Wrong lookups; actions taken in real systems Limit each tool to what the task needs; start read-only
Agent Plans and carries out several steps, using tools, with less human involvement between steps Errors compound across steps; outside content can steer actions Require a named person to approve anything that changes a record, reaches a patient or leaves the practice

Retrieval: ask three questions. Which documents does it search? Who adds, updates and removes them, and how often? Does every answer show the passage it used? If the answer to the last question is no, every answer has to be treated as recall and checked from scratch.

Tools and permissions: think in rungs. A useful way to picture the risk of a connected feature is a ladder. Module 12 develops it fully.

  1. Reads only: looks up information and shows it to a person.
  2. Drafts for review: prepares a message, form or task that waits for a person.
  3. Writes to a system: changes the schedule, the practice management system or a record.
  4. Acts outward: sends a message to a patient, submits something to a payer, or releases information.

Indirect prompt injection: a defender's view. You do not need to know how such content is written. You need to design so that it cannot do much harm.

  • Assume any document from outside the practice may contain text aimed at the AI.
  • Keep AI features that read outside content on the reads-only or drafts-for-review rungs.
  • Make the approval step a real review of the proposed action, not a click-through.
  • Report any AI output that proposes something odd, such as releasing records to an unfamiliar address, to the security official.

Human roles in the configuration. NIST's AI RMF 1.0 (January 2023, voluntary and being revised as of September 2026) says human-AI configurations can span "from fully autonomous to fully manual," and that human roles "need to be clearly defined and differentiated." It also notes that in some conditions the AI part of a human-AI interaction "can amplify human biases, leading to more biased decisions than the AI or human alone." For every connected feature, write down which steps the system takes, which a person takes, and where the person can stop it.

Know the value chain. Ask the vendor, in writing: which model or models the product uses, which other companies process the data, where it is hosted, and how you will be told if any of that changes. Module 03 explains why the answers matter for patient information, and Module 13 turns them into a vendor review.

Remember this

  • Retrieval, tools and agents each add a new way to fail on top of the model's own confabulation.
  • A retrieved passage must be opened and read; an answer can go beyond it.
  • Generative AI mixes data and instructions, so outside content can steer a feature that reads it (NIST AI 100-2 E2025).
  • Start on the lowest permission rung and require named approval before a record changes or anything leaves the practice.
  • Mitigations lack guarantees, so human control points stay in place whatever a vendor says.

Medical practice example

Larkspur Orthopedic Associates PLLC is a fictional six-surgeon group. Its operations lead, Priya Natarajan, is reviewing a new feature proposed by the practice's fictional software vendor. The feature reads incoming referral faxes, extracts the referring clinician, the patient's insurance and the requested service, and can add a scheduling task to the work queue. By default it is also allowed to send the patient a portal message inviting them to book. Priya maps the feature to the four building blocks. It is a model with retrieval over the fax inbox and two tools: one that writes to the work queue and one that messages patients. She places the queue tool on the writes rung and the messaging tool on the acts-outward rung. During a two-week trial using fictional test faxes prepared by the vendor, one test fax contains a line addressed to "the assistant" rather than to a person, asking for the patient's records to be sent to an outside email address. The feature does not send anything, because Priya had asked for the messaging tool to be switched off for the trial. It does, however, create a queue task titled "send records to requester." The scheduler reviewing the queue recognises that records requests do not come through referral faxes, rejects the task and reports it to the security official. Priya's decision for go-live is recorded in one page: extraction stays on, every queue task waits for a scheduler's approval, the patient messaging tool stays off, and any task that proposes releasing information is blocked and routed to the privacy official. She also asks the vendor, in writing, which model and hosting companies the feature relies on.

Responsible-use warning

Do not accept a connected AI feature with its default permissions. Defaults are often set for convenience, and a feature that reads faxes, emails or portal messages can be steered by the people who wrote them. Keep features that read outside content on the reads-only or drafts-for-review rungs, and require a named person to approve any change to a record or anything sent outside the practice. A vendor's assurance that the product is protected does not replace that approval, because NIST notes mitigations lack guarantees.

Human-review requirement

The practice manager or named administrative owner decides which rung each connected feature may use and records who approves its outputs. The security official reviews any feature that reads outside content, and receives reports of odd proposed actions. The privacy official approves any path by which patient information could leave the practice. Anything a feature proposes that carries clinical content, such as a message about results or symptoms, goes to the licensed treating clinician. The reviewer confirms approvals are real reviews and that no default permission was accepted unexamined.

Practical exercise

Work with a fictional feature description only. Do not connect any tool to real systems for this exercise.

  1. Write a short description of a fictional AI feature that reads an inbox (faxes, portal messages or emails) and can take at least two actions.
  2. Label which of the four building blocks it uses: model alone, retrieval, tools, agent.
  3. List every tool it can call, and place each on the ladder: reads only, drafts for review, writes to a system, or acts outward.
  4. For each tool, write the lowest rung that would still achieve the task.
  5. Identify who outside the practice writes the content the feature reads.
  6. Describe, without writing any example wording, what could happen if that content contained instructions aimed at the AI.
  7. Name the human control point that would stop that outcome, and the person who owns it.
  8. Write three questions for the vendor about its value chain: models, other companies that process data, and hosting.
  9. Decide which role the feature must route to if it proposes anything clinical, and which if it proposes releasing information.
  10. Record your decision on one page in your workbook, including the rung for each tool, the approver for each output, and the date you made the decision.

Knowledge check

  1. Why does a retrieval feature not guarantee that its answer reflects the document it found?

    Answer guidance: The model still writes the answer. It can mix the retrieved passage with its own predictions, go beyond what the passage says, or rely on a stale or wrong document, so the cited passage must be opened and read.

  2. According to NIST AI 100-2 E2025, who mounts indirect prompt injection, and why is it possible?

    Answer guidance: A third party, not the primary user, mounts it by manipulating resources the system reads. It is possible because generative AI models combine the data and instruction channels, and often the primary user is harmed.

  3. What are the four rungs of the permission ladder in this lesson?

    Answer guidance: Reads only; drafts for review; writes to a system; and acts outward, such as sending a patient message or submitting to a payer.

  4. Why should human approval stay in place even when a vendor says its product resists prompt injection?

    Answer guidance: NIST states that many mitigations are empirical and lack theoretical or provable guarantees, so a residual risk remains and a human control point is what stops a wrong output becoming a wrong action.

Sources and further reading