Lesson objective
Diagnose a weak AI draft by failure type, refine the prompt one change at a time against fictional cases with known answers, and avoid follow-up questions that push the tool toward the answer you expect.
Plain-language explanation
Why this matters now
The first draft from a prompt is rarely the one you keep. Refinement is normal. The problem is how it is usually done: someone reads a draft, feels it is wrong, and types "are you sure?" or "shouldn't that be covered under the endorsement?" until the draft says what they expected. That is not refinement. It is steering, and it can turn a model's uncertain answer into a confident wrong one.
Best practice (voluntary framework): NIST AI 600-1 Section 2.2 says language models "sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect," and that outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer." A follow-up that asks the tool to justify an answer will usually get a justification. It is not evidence. The profile also describes automation bias as "excessive deference to automated systems," and the reverse problem is just as real in refinement: a user who expects a particular answer and keeps prompting until the tool agrees.
This lesson covers refinement by one person working on one prompt. Module 03 covers how a prompt is tested and released into the agency's shared library.
A question to sit with
When you ask a tool "are you sure?", what new information have you given it about the policy?
What goes wrong
Arguing the tool into agreement. Each follow-up hints at the answer the user wants. The final draft matches the user's expectation, and the user takes that as confirmation.
Changing everything at once. A weak draft prompts a rewrite of the task, the constraints and the output format together. The next draft is better or worse, and nobody knows why.
Refining on one case. The prompt is adjusted until it produces a good summary of one policy, then used on fifty others it was never tried on.
Fixing the symptom. A draft invented a limit, so the user adds "don't invent limits." The real cause was that the endorsement carrying the limit was never supplied.
Continuing a long conversation. After many turns, earlier drafts, corrections and discarded documents are all still in the conversation. The tool may draw on any of them.
The practical answer
Step 1: diagnose the failure type before changing anything. The Seo et al. study of Korean reimbursement guidelines (Journal of Medical Internet Research, 2026) sorted 235 wrong model predictions into three patterns: information gap-filling (196), criterion misapplication (20) and false uncertainty (19). That was a study of health reimbursement, not US policies, and it applies here only by analogy, but the three categories are a useful way to read a weak agency draft. The fixes below are this programme's recommendation.
| Failure type | What it looks like in an agency draft | Likely cause | First fix to try |
|---|---|---|---|
| Gap-filling | A limit, exclusion or premium that is not in any supplied document | A document missing, or no permitted unknowns answer | Supply the missing document; strengthen the unknowns rule |
| Misapplication | A real provision applied to the wrong coverage part, location or vehicle | Documents unlabelled, or context header incomplete | Label sources; add schedule and location detail to the header |
| False uncertainty | "Cannot be determined" for a point the declarations clearly state | Constraint wording too broad, or document hard to read | Check the document quality; narrow the constraint |
| Format failure | Statements without sources; paragraphs instead of the table | Output format block vague | Give an exact table layout |
| Scope drift | Advice, recommendations or sales language | Task block too open | Restate the task and use and add the recommendation constraint |
Step 2: make one change, then rerun. Record what you changed. If the draft improves, you know why. If it does not, undo it.
Step 3: rerun on a small set of cases with known answers. Before relying on a refined prompt, run it on at least three fictional or cleared cases where you already know the correct answer, including one where the right answer is "cannot be determined." A prompt that performs well on the case you refined it on has only shown that it fits that case.
Step 4: ask neutral follow-ups, not leading ones.
| Leading follow-up (avoid) | Neutral follow-up (use) |
|---|---|
| "Shouldn't water backup be covered under END-1?" | "List every provision in END-1, with page numbers." |
| "Are you sure the deductible is $1,000?" | "Quote the exact wording and page for the deductible." |
| "That seems wrong, try again." | "For each statement, give the source and page, or label it cannot be determined." |
| "The client needs this to be covered; explain why it is." | Do not ask. Route the coverage question to the producer. |
A neutral follow-up asks the tool to point to the documents. Whatever it points to, you then check yourself.
Step 5: start a fresh session for a fresh run. When you have changed the prompt, run it in a new conversation with only the current documents, so earlier drafts and corrections cannot leak into the answer. Check your approved tool's own documentation for how it handles conversation history; it varies.
Remember this
- Diagnose the failure first: gap-filling, misapplication, false uncertainty, format failure or scope drift.
- Change one thing at a time and record what you changed.
- Test a refined prompt on several cases with known answers, including one that cannot be determined.
- Ask follow-ups that make the tool point to the documents, never follow-ups that suggest the answer.
- A tool's justification of its answer is another output; only the source document settles it.
Insurance example
At the fictional Larkspur Benefits Group, service representative Theo Brandt, licensed for personal lines, is using a renewal change list prompt on the fictional client Gwen Aldous's auto policy with Brambleton Mutual (fictional). The draft says comprehensive coverage on her second vehicle was removed at renewal. Theo expected no changes. His first instinct is to type "that's wrong, there were no changes, fix it." He stops and diagnoses instead. He asks a neutral follow-up: "For the comprehensive change on vehicle 2, quote the wording and give the source and page for both the expiring and renewal status." The tool cites the renewal declarations page 2. Theo opens page 2 and finds the comprehensive column for vehicle 2 is blank on the renewal. It is not a tool error: the carrier's renewal really does show no comprehensive on that vehicle. He then checks whether the tool has been getting other items right. He runs the same prompt, in fresh sessions, on three fictional known-answer cases the agency keeps, one of which has a change the documents cannot explain. On two cases the output is correct. On the third, the tool gives a reason for a deductible change, "due to updated rating factors," which no document states. That is gap-filling. Theo makes one change to the prompt, adding a constraint that no reason for a carrier change is given unless a supplied document states it, reruns all three cases, and gets the third right. He records the diagnosis, the change and the results, sends the change to the library entry owner, and passes the vehicle 2 finding to licensed producer Martin Hale, who contacts the carrier and the client about the removed coverage.
Responsible-use warning
Do not refine a prompt until it agrees with you. If you already believe the answer, every leading follow-up pushes the tool toward that belief, and its eventual agreement feels like confirmation when it is only compliance. The draft that surprised you may be the one that is right, as the source documents will show. Check the page before you argue with the output, and never keep a private improved version of a shared prompt.
Human-review requirement
The licensed producer who owns the account checks any finding a refinement surfaces against the source documents, not against the conversation that produced it. Where the refined prompt will be used again, the producer or library entry owner checks the known-answer results, confirms the change was one change and was recorded, and decides whether it goes into the shared library. Anything a refinement turns up about the client's coverage is the producer's to explain and act on.
Practical exercise
Use fictional accounts and documents only.
- Write three fictional declarations pages for the same line of business, with invented carriers and form numbers.
- Build a known-answer sheet: five items per page, with at least one item on one page not settled by the documents.
- Write a deliberately weak prompt for a renewal change list or coverage summary.
- Write, as if a tool had produced it, a draft from that weak prompt containing one example each of gap-filling, misapplication and false uncertainty.
- Label each problem in the draft with its failure type from the table in this lesson.
- For each problem, write the one prompt change you would try first, and explain in one sentence why it addresses the cause rather than the symptom.
- Rewrite three leading follow-up questions you might naturally ask as neutral follow-ups that make the tool point to the documents.
- Describe how you would rerun the changed prompt on all three pages, including starting fresh sessions.
- Write the refinement note for the account AI use record: failure seen, change made, known-answer result.
- Draft a two-sentence proposal to the library entry owner describing the change.
- Note one thing your known-answer check could not tell you about the prompt.
Knowledge check
-
What is the difference between refining a prompt and steering the answer?
Answer guidance: Refining diagnoses a failure, changes the prompt once, and checks the result against known answers and the source documents. Steering uses follow-ups that hint at the expected answer until the tool agrees.
-
Which three failure patterns did Seo et al. identify, and how are they used here?
Answer guidance: Information gap-filling (196 of 235 errors), criterion misapplication (20) and false uncertainty (19). The study concerns Korean reimbursement guidelines, so the categories are used by analogy to diagnose weak agency drafts.
-
Why test a refined prompt on several known-answer cases?
Answer guidance: Because a prompt adjusted on one case has only shown it fits that case. Several cases with known answers, including one that cannot be determined, show whether the change helps generally or just hides the original error.
-
Why is a tool's explanation of its answer not evidence that the answer is right?
Answer guidance: NIST AI 600-1 notes that models sometimes give logical steps for incorrect answers and may produce confabulated reasoning or citations. Only the source document settles the point.
Sources and further reading
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- Seo J, Kim T, Kim JH; Journal of Medical Internet Research — Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study (2026-06-15). Accessed 2026-09-15.
- Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani; Organization Science — Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (2026-03-01). Accessed 2026-09-15.
- AI Coalition Network — Certification Educational Standard: Human Review of AI-Assisted Professional Work. Accessed 2026-09-15.