Lesson objective
Explain how generative models produce copy and images, and use that explanation to predict where a draft is likely to be wrong.
Plain-language explanation
Why this matters now
Generative tools are the part of AI marketers meet most directly, because they produce the deliverable itself: the headline, the product description, the press release, the storyboard frame, the voice-over. An NBER working paper by Bick, Blandin and Deming (September 2024, revised February 2025) reports that, as of late 2024, nearly 40% of the U.S. population aged 18 to 64 used generative AI. That figure covers everyone, at work and at home, not marketers, and it comes from a paper that has not been peer reviewed. It is still enough to say that clients, audiences and colleagues are now used to seeing generated content.
You do not need to be an engineer to use these tools well. You do need a working picture of how they produce output, because that picture tells you where to look for mistakes.
A question to sit with
An assistant writes a confident paragraph saying your client's product "won a national design award in 2024." Where did that sentence come from, if nobody gave the assistant that fact?
What goes wrong
Believing the model looked something up. A general assistant with no connected source does not consult a database of facts about your client. It produces likely-sounding text. An award, a statistic, a quote or a date can appear because it fits the pattern, not because it is true.
Trusting the explanation because it sounds reasoned. Ask a model why it chose a claim and it will give reasons. NIST notes that language models "sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect." The reasons are generated the same way as the answer.
Forgetting that the model's knowledge stops. A model is trained on data up to a point in time. Prices, platform rules, laws and a client's product range can all have changed since then.
Mixing instructions and material without noticing. When you paste a customer review, a competitor's web page or an email into a prompt, the model receives that text in the same channel as your instructions. Text inside the pasted material can be read as an instruction.
Treating a generated image as a photograph. An image generator produces a picture that fits a description. A product shown in it may have features the real product lacks, and a person shown in it may resemble a real person.
The practical answer
How a language model writes. A large language model is trained on a very large amount of text to predict what comes next. It works in small units called tokens, which are whole words or pieces of words. Given your prompt, it predicts a likely next token, adds it, and repeats, building the answer one token at a time. Three consequences follow for marketing work.
- Likely is not the same as true. The model is choosing what tends to follow, not checking facts. That is why it can produce an award, a percentage or a quote that no one supplied.
- The same prompt can give different answers. Most tools add some variation on purpose, so two runs can differ. Checking one run does not check the next.
- What you supply beats what it remembers. A model given your client's approved claims and told to use only those is on firmer ground than one asked to recall facts. It can still add or alter facts, so you still check.
NIST's Generative Artificial Intelligence Profile (NIST AI 600-1, 26 July 2024, a voluntary framework) calls false, confidently stated output confabulation, "known colloquially as 'hallucinations' or 'fabrications'", and says confabulations "are a natural result of the way generative models are designed." It adds that outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer." NIST does not give a rate for how often this happens, so do not attach one.
How an image, video or voice generator works, in outline. These systems learn from very large collections of media, usually paired with descriptions, and produce new media that fits the description you give. The same lesson applies: the output fits the request, which does not make it accurate. For marketing that means a generated packshot may not match the real product, a generated scene may include text or logos you did not ask for, and a generated face or voice may resemble a real person. Module 9 covers rights and likeness, and Module 10 covers labels and provenance.
Data and instructions share one channel. NIST's Adversarial Machine Learning taxonomy (NIST AI 100-2 E2025, March 2025, voluntary) explains that generative AI models "combine the data and instruction channels." It describes indirect prompt injection, where the attack is mounted "not by the primary user of a model but instead by a third party" who places text in a resource the system reads, and notes that often the primary user is the one harmed. For a marketer, the resource might be a review, a web page or an inbound email. The defender-side habit is simple: treat anything pasted in as material to be worked on, not as instructions, and never let a tool act on such material without a person's approval.
Use this table to predict where a draft is weakest.
| Part of the output | Why it is at risk | Check |
|---|---|---|
| Facts about the client or product | Generated from patterns, not looked up | Compare with the approved claims file |
| Figures and percentages | Plausible numbers are easy to generate | Trace every figure to its source; recompute |
| Quotes and testimonials | May be invented or merged | Use only quotes you hold in writing, with permission |
| Sources and citations | May be confabulated | Open each source and confirm it says what is claimed |
| Current rules, prices, dates | Training stops at a date | Check the current primary source |
| Anything following pasted material | Material may carry instruction-like text | Read the output for content you did not ask for |
| Generated images and audio | Fit the description, not reality | Compare with the real product; check for real-looking people, logos and text |
Supplying facts, in practice. Module 6 builds a full prompt structure. For now, adopt one habit: give the model the material it must rely on (the approved claims, the brand rules, the product facts) and tell it to use only that material and to mark anything it cannot support. Then check that it did.
Remember this
- A language model builds text one token at a time by predicting what is likely, not by looking facts up.
- NIST calls confident false output confabulation and says it is a natural result of how generative models are designed.
- Reasons a model gives are generated too; they are not proof.
- Data and instructions share one channel, so pasted material can steer the output.
- Generated images, video and voice fit the request, not reality; compare them with the real thing.
Marketing agency example
Kestrel Media Partners, a fictional agency, is preparing launch copy for Hollin Bikes, an invented maker of electric cargo bikes. Copywriter Tomás Reyes gives a general assistant the product name and asks for a 150-word product page. The draft is lively and reads well. It says the bike has "a 90-mile range on a single charge", "won a national design award in 2024" and is "trusted by over 10,000 families". Tomás checks each against the client's approved claims file. The file gives a range of 55 miles under stated test conditions, mentions no award, and records 3,200 bikes sold, not 10,000 families. All three sentences were generated because they fit the pattern of bike launch copy. He asks the assistant why it chose 90 miles, and it explains that "most premium cargo bikes offer 80 to 100 miles," which is itself unsupported. Tomás starts again. This time he pastes the approved claims file into the prompt and instructs the assistant to use only those facts and to mark anything it cannot support. The second draft uses the 55-mile figure with its test conditions and leaves out the award. It still says "loved by thousands of riders", which the file does not support, so Tomás replaces it with the approved sales figure. He also generates three lifestyle images. One shows the bike with a second battery the real model does not have; he rejects it. His review note lists each claim, its source in the file, and the rejected image, and goes to creative director Hana Lowe before anything is sent to Hollin Bikes.
Responsible-use warning
Do not ask a general assistant to "fill in" product facts, awards, statistics, testimonials or sources and then assume a fluent answer is accurate. The model produces likely text, not verified facts, and its explanation of a choice is generated the same way. Supplying approved material reduces the risk but does not remove it, so every claim still needs checking. Never accept a generated image as an accurate depiction of a real product without comparing it to the product.
Human-review requirement
The named agency reviewer, usually the creative director or account lead, checks every fact, figure, quote, source and product depiction in generated content against material the agency holds, and rejects anything unsupported, including reasons the model offered for its choices. The reviewer confirms the draft was built from supplied, approved material where possible and that pasted outside material did not steer the result. The client decides whether its claims run, and counsel answers questions about what a claim legally requires.
Practical exercise
Use fictional material only. Do not paste real client or customer information into any tool.
- Invent a simple product: name, three true features, one test figure with its conditions, and a sales figure. This is your approved claims file.
- In an approved tool, ask for a 100-word product description giving only the product name and category.
- Highlight every fact, figure, award, quote and source in the draft.
- Mark each one "in the file", "different from the file" or "not in the file".
- Ask the tool why it included one unsupported item, and note whether the reason is itself supported.
- Run the request again, this time pasting your claims file and instructing the tool to use only that material and to mark anything it cannot support.
- Repeat steps 3 and 4 on the second draft and compare the counts.
- Run the second prompt once more and note whether the output changed.
- Write three sentences on what the comparison shows about supplying facts versus asking the model to recall them.
- If your tool can generate images, ask for one image of your invented product and list every feature, word or logo in it that your claims file does not describe.
- Write the review note a creative director would need: each claim, where it came from, and what you removed or changed.
Knowledge check
-
Why can a language model produce an award or statistic that nobody gave it?
Answer guidance: It generates text by predicting likely next tokens from patterns in training data. An award or figure can appear because it fits the pattern of similar copy, not because it was looked up or is true.
-
What does NIST AI 600-1 say about confabulation?
Answer guidance: It defines confabulation as confidently stated but erroneous or false content, says it is a natural result of how generative models are designed, and notes outputs may include confabulated logic or citations. It gives no rate.
-
What does it mean that generative models combine data and instruction channels?
Answer guidance: Text you supply as material, such as a pasted review or web page, arrives in the same channel as your instructions, so instruction-like text inside it can steer the output. NIST describes third parties exploiting this through indirect prompt injection.
-
Why should a generated product image be compared with the real product?
Answer guidance: An image generator produces media that fits the description, not an accurate record, so it may show features, text or logos the real product does not have.
Sources and further reading
- National Institute of Standards and Technology — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024-07-26). Accessed 2026-09-15.
- National Institute of Standards and Technology — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025). Accessed 2026-09-15.
- National Bureau of Economic Research (NBER) — The Rapid Adoption of Generative AI (NBER Working Paper 32966) (2024-09-01). Accessed 2026-09-15.