Lesson objective
Design the escalation that fires the instant an intake exchange turns clinical, and state why a patient-facing assistant never answers a symptom, medication or urgency question.
Plain-language explanation
Why this matters now
Intake conversations do not stay administrative because the practice designed them that way. A patient completing a registration form types, in the free-text box meant for "reason for visit", that she has had chest tightness since Thursday and is not sure whether to wait for the appointment. A parent replying to a form link asks whether to keep giving a medicine that seems to be causing a rash. Neither person is misusing the form. They are telling the practice something, using the channel the practice opened.
What happens in the next sixty seconds is a design decision the practice made months earlier.
Recommended practice (ECRI, published 21 January 2026; independent nonprofit guidance, voluntary): ECRI ranked "Misuse of AI chatbots in healthcare" first in its Top 10 Health Technology Hazards for 2026. ECRI says large language model chatbots "are not regulated as medical devices nor validated for healthcare purposes" but are increasingly used by clinicians, patients and health care personnel. Its experts report chatbots suggesting incorrect diagnoses, recommending unnecessary testing, promoting subpar medical supplies and inventing body parts; in one ECRI test a chatbot incorrectly said an electrosurgical return electrode could be placed over the shoulder blade, which would risk burns. ECRI recommends verifying chatbot information with a knowledgeable source, and that health systems establish AI governance committees, provide clinician AI training and regularly audit AI tools' performance. ECRI's list is not a regulation and binds no practice. Second on the same list is unpreparedness for a "digital darkness" event, and ninth is health technology implementations that prompt unsafe clinical workflows.
This level's position is simpler than ECRI's general warning, because it applies to a specific thing: a patient-facing assistant in a practice never answers a symptom, medication or urgency question. Not with a hedge, not with a disclaimer, not "in general". It hands off.
A question to sit with
When a patient types something worrying into your intake form at 6 pm on a Friday, what does your practice do next, and how does that patient know what to expect?
What goes wrong
The assistant answers helpfully. It offers general information about chest tightness. It is well written, and it is a clinical statement made by software to a patient.
The reassurance nobody wrote. "Your appointment is on the 14th" in reply to "should I wait?" reads as clinical reassurance even though no clinical claim was made.
The escalation lands in a queue that sleeps. The trigger fires correctly and creates a task in a work list nobody watches after five o'clock.
The words become the trigger. A keyword list catches "chest pain" and misses "tightness across my front when I walk upstairs".
The paraphrase replaces the patient. Clinical staff read a tidy one-line summary rather than what the patient wrote, and the summary drops the detail that mattered.
Nobody tells the patient. The exchange escalates silently, so the patient waits, believing the practice has answered.
The practical answer
Write the escalation rule in terms of subject, not sentiment. AI Coalition Network recommendation: an intake exchange stops being administrative and escalates the moment it touches any of these:
| Trigger | Examples in a practice |
|---|---|
| A symptom or physical complaint | Pain, bleeding, breathlessness, a new or changing lump, a rash |
| A medication question | Whether to take, stop, change or continue anything |
| Urgency | Should I be seen sooner, should I go to the emergency department, can this wait |
| A result | Any question about a test, image or laboratory value |
| Pregnancy, an infant, or a recent procedure | Any context where the same complaint carries different weight |
| Distress or safety | Anything suggesting self-harm, harm from another, or a person unable to manage at home |
Escalate on the whole message, not on keywords. A keyword list is a floor, not a design. Route the patient's own words, unedited, to the receiving person. Where the automation summarises, show the summary beside the original rather than instead of it, and mark any message the automation could not read fully.
Name the receiver and the clock, per channel and per hour. An escalation without a named receiver is a queue. Write, for each channel, who receives an escalation inside working hours, who receives it outside them, what the patient sees in the meantime, and what happens if nobody has acted within the practice's own stated time. The practice sets that time; no source recorded in this programme sets one for a practice.
Tell the patient, in the same breath. The automation's only clinical-adjacent output is a hand-off message: it says that a person will respond, says roughly when, and says what to do if the problem worsens, using the practice's standard after-hours and emergency instruction written and approved by clinicians. That instruction is drafted by clinicians once, stored as fixed text, and never generated afresh by a model.
Know what a patient-facing assistant is allowed to say at all. The safe set is narrow: appointment times, location and parking, what to bring, how to complete a form, how to upload a document, how to reach a person. Everything else is a hand-off.
A state example, and only one state. Law (Utah): S.B. 226 (2025 General Session), enacting Utah Code Title 13, Chapter 75, effective 7 May 2025. Section 13-75-103(2) says an individual providing services in a regulated occupation shall "prominently disclose" when a recipient is interacting with generative AI, if that use "constitutes a high-risk artificial intelligence interaction". Under 13-75-101(5) a high-risk interaction includes collecting sensitive personal information such as health data, and includes personalised recommendations someone could rely on for significant personal decisions, including "medical advice or services" and "mental health advice or services". Section 13-75-103(3) says the disclosure is given verbally at the start of a verbal interaction and in writing before a written interaction. Section 13-75-101(8) defines a regulated occupation as one regulated by the Department of Commerce requiring a licence or state certification. Section 13-75-102 says it is no defence to a consumer protection violation that generative AI made the statement or carried out the act. The division director may impose an administrative fine of up to $2,500 per violation under 13-75-105(4)(a).
Three limits. This is Utah only. Whether a given medical practice role counts as a regulated occupation under the statute was not verified for this programme, so do not tell learners that all Utah physicians are covered. And "generative artificial intelligence" is defined narrowly in the statute: a trained, conversational system producing non-scripted outputs with limited or no human oversight. A practice in another state checks its own law; several states have AI disclosure statutes with different triggers, and Level 4 mapped them.
One design point follows from the Utah drafting regardless of where you practise. A scripted, fixed-text assistant that does not generate its answers is a different thing from a conversational one, in the statute's own definition and in risk. If all a patient-facing assistant needs to do is confirm appointments and collect form answers, fixed text does the job and removes the hardest problem.
Remember this
- A patient-facing assistant never answers a symptom, medication or urgency question, with or without a disclaimer.
- Escalate on subject matter, route the patient's own words, and never let a summary replace what the patient wrote.
- Name the receiver and the clock for every channel and every hour of the day, and tell the patient a person is coming.
- The after-hours and emergency instruction is fixed text written by clinicians, never generated fresh.
- ECRI put chatbot misuse first for 2026 and says chatbots are not regulated as medical devices nor validated for healthcare purposes; its guidance is voluntary. Utah's disclosure duty is a state example with unverified reach into practice roles.
Medical practice example
Meridian Street Dermatology, a fictional three-clinician practice, runs a scripted intake assistant on its website that books appointments and collects registration answers. Six weeks in, the practice manager, Kofi Adjei, audits a week of transcripts with the clinical lead. Eleven of 214 exchanges crossed out of administrative territory. Nine were straightforward: patients describing a changing mole, a spreading rash, a reaction to a cream. The assistant escalated all nine correctly and told each patient that a nurse would call. Two did not go well. In the first, a patient wrote that a prescribed cream was "burning quite badly" and asked whether to keep using it. The assistant, working from a fixed answer about medication questions, replied that the practice could not advise on medicines by message and that a nurse would call within one business day. It was 5.40 pm on a Friday. The escalation landed in a work list checked at 8 am Monday, and the patient stopped the cream on her own and waited the weekend in discomfort. In the second, a patient described a lesion that had bled twice and asked whether it could wait until his appointment in three weeks. The assistant escalated, but the summary given to the nurse read "patient asks about appointment timing", and the bleeding did not appear in the one-line summary. The nurse, reading only the summary, replied about scheduling. Kofi makes two changes. Every escalation now carries the patient's own words above any summary, marked as the patient wrote them, and the summary is labelled as a summary. And the practice writes an out-of-hours rule: escalations raised after 4 pm on a weekday or at any time at a weekend trigger a message telling the patient that a person will respond on the next working day, together with the practice's standard instruction, written by its clinicians, on what to do if a problem worsens before then. Kofi also asks the vendor to confirm, in writing, that the assistant cannot generate free-text clinical content under any configuration, and records the answer in the authority register.
Responsible-use warning
A disclaimer does not convert a clinical statement into an administrative one. "This is general information and not medical advice, but chest tightness can have many causes" is a clinical statement made by software to a patient, whatever follows the comma. The safe set for a patient-facing assistant is administrative: appointments, location, what to bring, how to complete a form, how to reach a person. Everything else is a hand-off to a named person, with the patient told that a person is coming and roughly when.
Human-review requirement
The licensed treating clinician, or the clinical lead acting for the practice, writes and approves the fixed hand-off text, including the after-hours and emergency instruction, and approves the subject-matter triggers that route to clinical staff. The practice manager or named administrative owner owns the escalation routes, the named receivers for each channel and hour, and the audit of transcripts. Both review the trigger list after any escalation that arrived late or went to the wrong place, and record the date and the change.
Practical exercise
Use a fictional practice or your own, with no real patient messages.
- List every channel through which a patient can send your practice text before an appointment: web assistant, portal message, form free-text box, text message, email, online booking notes.
- For each channel, write who receives an escalation during working hours and who receives it outside them, by role.
- Write your escalation trigger list by subject matter, starting from the six rows in the lesson and adding any your specialty needs.
- Write the fixed hand-off text a patient receives, including what to do if the problem worsens. Have a clinician approve it, and record that approval.
- Write ten fictional patient messages that should escalate, using ordinary words rather than clinical terms, and ten that should not.
- Run all twenty past your current design on paper. Record every miss in both directions and what the patient would have experienced.
- Check whether any escalation route ends in a queue nobody watches at 6 pm on a Friday. Name the person who watches each one.
- Decide whether your patient-facing assistant generates text at all, or works from fixed answers, and write the reason in the authority register.
- Look up your own state's AI disclosure law, record what you found and the date you checked, and write one sentence on what it does and does not reach.
- Write the sentence a front-desk staff member says when a patient asks whether they were talking to a real person.
Knowledge check
-
What did ECRI rank first in its Top 10 Health Technology Hazards for 2026, and what status does that list have?
Answer guidance: It ranked misuse of AI chatbots in healthcare first, in a press release dated 21 January 2026, noting that such chatbots are not regulated as medical devices nor validated for healthcare purposes. ECRI is an independent nonprofit and its list is voluntary guidance that binds no practice.
-
Why is escalating on keywords alone an inadequate design?
Answer guidance: Because patients describe problems in ordinary words. A list that catches "chest pain" misses "tightness across my front when I walk upstairs". The trigger should be the subject matter of the whole message, and the patient's own words should reach the receiving person unedited.
-
What does Utah Code 13-75-103(2) require, and what are its limits?
Answer guidance: An individual providing services in a regulated occupation must prominently disclose when a recipient is interacting with generative AI if the use is a high-risk interaction, verbally at the start of a verbal interaction and in writing before a written one. It applies in Utah only, and whether a given practice role is a regulated occupation was not verified here.
-
Why does this course give no triage criteria?
Answer guidance: Because clinical urgency belongs to the licensed treating clinician, or to clinical staff under an arrangement the clinician owns, and it sits on the never list. The subject-matter triggers decide only that a clinician looks at the message; they say nothing about how urgent anything is.
Sources and further reading
- ECRI — Misuse of AI chatbots tops annual list of health technology hazards (2026-01-21). Accessed 2026-09-15.
- Utah Legislature — S.B. 226 (2025 General Session), Artificial Intelligence Consumer Protection Amendments, enacting Utah Code Title 13, Chapter 75 (2025-05-07). Accessed 2026-09-15.
- OWASP Foundation, OWASP GenAI Security Project — OWASP Top 10 for LLM Applications 2026 (2026-08-04). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.