Reliable output
Why AI makes things up, and how to stop it
A customer rings to ask about the 20% discount your website chat offered them yesterday. You have never offered a 20% discount, and now you have to decide whether to honour it.
Why does AI make things up?
AI makes things up because it is built to produce a likely answer, not a checked one. It learned the patterns of language from a huge amount of text, and it writes by predicting what usually comes next. When the real fact is missing, the pattern still produces something that sounds right.
People call this hallucination. The word makes it sound rare and strange. It is neither. It is the normal behaviour of the tool when it is asked something it can't see. Researchers have studied it for years: a widely cited survey of hallucination in language generation, published in ACM Computing Surveys, describes it as a known weakness of this whole family of systems rather than a fault in one product.
It predicts words, it doesn't look them up
Ask a model your opening hours and, unless it has been given them, it has nothing to look up. It knows what opening hours for a business like yours usually look like, so it writes those. Nine to five sounds right for a bookkeeper, and that is exactly the problem.
It sounds just as sure when it's wrong
A person who is guessing usually sounds like it. AI doesn't. A made-up price and a real one come out in the same calm, tidy sentence, which is why invented answers get past busy people.
It has been rewarded for guessing
Models are trained to give useful answers, and an empty answer looks unhelpful. A 2025 paper, Why Language Models Hallucinate by Kalai and colleagues, argues that the way models are trained and scored rewards guessing over admitting uncertainty, much like a multiple-choice exam where a blank always scores zero. So asked about a refund policy it was never told, a model will often supply one, unless it has been told clearly that "I don't know" is the right answer.
How often it happens depends on the job
There is no single rate, because it depends on the task and what the AI can see. One useful warning comes from law, where answers are hard and checkable. Stanford HAI reported that legal research tools built to cite their sources still gave incorrect information more than 17% of the time in its tests, and one more than 34%. If specialist tools with sources do that on hard questions, a general chatbot guessing about your business can too.
Where made-up answers hurt a small business
The damage is worst wherever the AI states something specific about your business to a customer. A vague sentence costs nothing. A wrong number, date or promise costs money or trust, and often both.
Prices and discounts
An invented price is the classic case. Too low and you either honour it or disappoint someone. Too high and you lose the job without knowing why.
Dates, hours and availability
A venue's chat saying a date is free when it isn't, or a café's assistant giving last year's Christmas hours. These look like small slips and create real problems on the day.
Policies and promises
Refunds, guarantees, cancellation terms. If the AI states a policy you don't have, a customer has it in writing and expects you to stand by it. For services, the Consumer Rights Act 2015, section 50 says that what is said or written to a customer on a trader's behalf can become a term of the contract if they rely on it. That is a good reason to treat your chat window as a member of staff, not a toy.
Facts about people
An AI summary that gets a customer's history wrong, or mixes up two patients with the same surname, is more than embarrassing. If it is personal data, the accuracy rules apply. The ICO's guidance on accuracy and statistical accuracy in AI says AI outputs need not be right every time, but you must allow for the chance of error and its effect on decisions you make from them.
What AI assistants say about you elsewhere
Made-up answers are not only a problem inside your own tools. When a customer asks ChatGPT or Gemini about your business and it can't find clear facts, it may guess your prices or hours. Our post on getting found in AI answers covers how to give those assistants something real to quote.
How to keep made-up answers away from customers
- 01
Give it your real facts in writing
Most invention happens because the AI wasn't given the answer. Put your prices, hours, services, areas and policies in one document it can read, and keep that document current. An AI that can see the fact rarely needs to invent it. If your facts are scattered across old emails and folders, a Knowledge Index maps every document and shows which ones have gone stale.
- 02
Tell it that "I don't know" is a good answer
Write it into the instructions: if the answer isn't in the document, say you'll check and pass it to a person. Reward that answer when you see it. Anthropic's own guide to reducing hallucinations puts this first, and says giving the model explicit permission to admit uncertainty can sharply reduce false information. Our guide to writing instructions an AI will follow shows how to word it.
- 03
Keep it to one job
An assistant that only answers booking questions has far fewer chances to invent something than one that will chat about anything. A narrow job means a short list of facts it needs and a short list of things to check. Tell it, in writing, to use only the documents you gave it and not its general knowledge.
- 04
Make it say where each answer came from
Ask it to quote or name the part of your document each answer is based on. If it can't point to the source, treat the answer as a guess and have it remove the claim. This makes the checking much quicker too, because you compare a quote with a page instead of searching for the fact yourself.
- 05
Check anything specific before it leaves
Numbers, dates, names and promises get checked against your records before they reach a customer, by a simple automatic check or by you approving a draft. Our post on checking AI's work before a customer sees it shows how to set that up for one job.
- 01Your facts, written downprices, hours, policies in one current document
- 02"I don't know" allowedthe instructions say a missing fact goes to a person
- 03One narrow jobfewer questions it could guess at
- 04A source for every claimno quote, no claim
- 05A check before it sendsnumbers, dates and promises compared with your records
A worked example: a dental practice's website chat
Here is how those steps play out on one job. The practice and its details are an example, but every problem in it is a common one.
A two-dentist practice in Bristol adds a chat box to its website to answer common questions out of hours. In the first week, it tells one patient an emergency appointment costs £45 and another that the practice is open on Saturdays. Neither is true.
Why it happened
The chat had been given the practice's website pages and nothing else. The fees page listed routine treatments but not emergency appointments, and nothing anywhere mentioned Saturdays either way. Asked about both, the AI filled each gap with what dental practices often charge and often do.
What the practice changed
The practice manager wrote a one-page fact sheet: every fee, the opening hours including the days it is closed, and what happens out of hours. The instructions gained two lines: answer only from the fact sheet, and if the answer isn't there, say a member of the team will reply in the morning. Anything about fees now ends with "the team will confirm your exact cost".
What happened next
Over the next few weeks, the share of chats handed to the team in the morning went up, and the made-up answers stopped showing up in the weekly review. That is the right trade. A chat that says "we'll check" a bit more often costs a few minutes of the manager's morning. A chat that invents a price costs a patient's trust.
How to spot an answer that might be made up
You can't always tell by reading, but some answers deserve a second look every time. Treat these as warning signs when you review AI work, or build them into your checks.
A precise number with no source
"Most boiler services take 47 minutes" sounds researched. If the AI can't point to where 47 came from, it probably didn't come from anywhere. Precision is not the same as accuracy.
A reference you can't find
Links, product codes, case names, regulations, quotes from people. Made-up references look exactly like real ones. Open every link and search every name before it goes to a customer.
An answer it couldn't know
Today's stock level, whether a date is free, what a supplier will charge next month. If the AI has no live connection to that information, any confident answer is a guess.
Two different answers to the same question
Ask the same question twice in fresh chats. If the answers disagree on a fact, at least one is invented. Anthropic's guide lists this comparison as a way to catch hallucinations, and it costs you a minute.
Will newer AI models stop making things up?
Newer models make things up less often, but none of them has stopped completely, and you shouldn't plan as if one will. The way they work, predicting likely words, means a gap can always be filled with something plausible. Better models make the problem rarer, not gone. Even Anthropic's guide says its techniques reduce hallucinations but don't eliminate them, and tells readers to always validate critical information.
That is why the set-up matters more than the model here. Your facts written down, a narrow job and a check before anything leaves will keep working whichever model sits underneath, and they keep working when you switch. The models also change under you: the National Cyber Security Centre warns, in Exercise caution when building off LLMs, that providers can update a model in ways that break what worked before, and advises designing for the worst case. Our post on what AI can't do for your business yet covers the other limits worth planning around.
Questions
07Asked and answered
An AI hallucination is when an AI states something false as if it were true, usually because it didn't have the real fact and produced a likely-sounding one instead. It is normal behaviour for these tools, not a rare fault. The defence starts with checking specifics before they leave, which we cover in how to check AI's work before a customer sees it.
No, but you can keep made-up answers away from customers. Give it your facts in writing, let it say "I don't know", keep it to one job and check numbers, dates and promises before they go. Each layer catches some of what the others miss. Our safety page sets out what AI never does without a person.
Every leading model still makes things up sometimes, and the models change often, so the set-up around the model matters more than which one you pick. A model with your facts and a narrow job beats a better model guessing. We compare the main options in which AI model a small business should use.
Yes. If AI assistants can't find clear facts about you, they may guess your prices or hours, or leave you out of the answer altogether. The fix is giving them something real to quote. An AI Visibility Audit shows whether ChatGPT, Claude and Gemini cite you, name you or miss you, with a fix list.
Connect the assistant to your real services, prices and availability, so it reads them instead of guessing. Customers then get your actual answer rather than a likely one. That is what the AI Assistant Connector does, and AI Search Readiness sets your website up so Google and AI assistants can read, trust and quote it.
Give it one fact document, say plainly that "I don't know, I'll check" is the right answer when a fact is missing, and tell it to use only what you gave it. Add an example of each kind of answer. Our guide to writing instructions an AI will follow goes through it step by step.
That depends on the situation and is a question for a legal adviser, not this post. The safer path is to stop it happening: give the AI your price list, tell it to confirm prices with a person, and keep someone approving any price before it reaches a customer, as described on how it works.
This is general guidance, not legal or financial advice. For your own situation, speak to a qualified adviser.