Reliable output
How to check AI's work before a customer sees it
The AI has drafted forty replies overnight and they all read well. Before any of them goes out, you need to know which ones are wrong, and you haven't got time to read forty emails properly.
What should you check before AI work goes out?
Check four things: the facts it states, the promises it makes, whether it sounds like you and is addressed to the right person, and anything it added that you never asked for. Most AI mistakes at work fall into one of those four, and each one has a quick way to spot it.
You don't need to read every word the same way. Read for these four, in this order, and you will catch most of what matters in a fraction of the time.
The facts it could have got wrong
Prices, dates, opening hours, stock, job details. These are the mistakes that cost money and the ones customers notice first. If a reply names a number, that number needs to match a record you can point to. A plumber's quote that says £180 when the price list says £210 is not a typo. It is a lost £30 or an awkward phone call.
Wrong facts usually appear because the AI filled a gap it couldn't see past. Our post on why AI makes things up explains where that comes from and how to starve it of gaps.
The promises it made
AI is eager to help, and helpful sounds like "we can definitely fit you in tomorrow" or "we'll refund that straight away". Look for anything that commits you: a date, a refund, a discount, a guarantee. If you didn't tell it it could promise that, it shouldn't have.
This matters more than it looks. Under the Consumer Rights Act 2015, section 50, what is said or written to a customer by or on behalf of a trader about a service can be treated as a term of the contract if the customer relies on it. A reply sent from your inbox is sent on your behalf, whoever drafted it.
The name, the tone and the person
Is it to the right customer? Does it use their name correctly? Would you say it to them face to face? A café owner who writes short, warm replies will look odd sending three formal paragraphs. Customers notice when you stop sounding like yourself.
Anything it wasn't asked for
Extra paragraphs, an attachment, a new offer, a link to a page you don't have. AI fills space. The safest reply is often the shortest one, so treat anything extra as a question rather than a bonus.
Be strictest with marketing. If the AI writes a social post or advert that says "the fastest boiler service in Leeds", the advertising rules apply to you, not the tool: the CAP Code, section 3 says marketers must hold documentary evidence for claims customers are likely to see as objective.
How to set up checks for one job
- 01
Write down what a good result looks like
Take one job, say replying to booking enquiries, and write half a page: what a good reply includes, how long it is, what it never says. Add two real examples you were happy with. This is the standard every check measures against, and the same page doubles as the AI's brief. Our guide to writing instructions an AI will follow shows how to turn it into standing instructions.
- 02
List the mistakes you would hate to see
Not every possible mistake. The five or six that would actually hurt: the wrong price, a date you can't do, a customer's name spelt wrong, a promise of a refund. This list becomes your checks.
- 03
Let a machine check the fixed things
Anything with a right answer can be checked automatically. Does the price in the reply match the price list? Is there a name in the greeting? Do the links work? Does it contain words you have banned? A simple check runs the same way every time and never gets tired.
- 04
Read the rest yourself, before it sends
Judgement stays with you: tone, whether the answer actually helps, anything awkward. Set the job up so the AI writes drafts and you approve them. Approving a good draft takes seconds; writing one from scratch takes minutes.
- 05
Turn every catch into a check
When you find a mistake, fix the reply, then ask what would have caught it. Add that rule to the instructions or that check to the list. The same mistake should never reach you twice. We set out the habit in every AI mistake should become a rule.
A worked example: a heating engineer's Monday inbox
Here is what the two layers look like on a real kind of morning. The figures are an example, not a measurement, but the shape is typical of a small trade business.
A heating engineer in Sheffield gets twelve enquiries over the weekend. The AI drafts a reply to each one from the price list, the diary and a half-page brief. Before 8am, the checks have run.
What the checks catch on their own
Two drafts quote a boiler service at last year's price, because the price list says one thing and an old email the AI read says another. The price check flags both. One draft offers Thursday, which the diary shows as full. One has no name in the greeting because the enquiry came from a web form with the name field left blank. None of these needed a person to spot them. They needed a rule and a record to compare against.
What the owner still reads
That leaves eight clean drafts and four flagged ones. The owner reads the flagged four properly, fixes them, then reads the eight clean ones for tone. One of the eight is to a customer who complained last month, and the cheerful reply is wrong for them. No check knew that. The owner did, in about ten seconds.
What changes for next Monday
The old email gets removed from what the AI reads, so the stale price can't come back. A new rule says: if the name is missing, open with "Hello" rather than guessing. And customers with an open complaint get a tag, so their drafts always land in the flagged pile. Three catches, three new rules. Next Monday's reading is shorter.
Which check catches which mistake
| Mistake | Check that catches it | Who runs it |
|---|---|---|
| Wrong or old price | Compare every number with the current price list | Machine |
| A date you can't do | Compare any date offered with the diary | Machine |
| Wrong or missing name | Greeting must match the name on the enquiry | Machine |
| Broken or made-up link | Open every link before it sends | Machine |
| Banned words or promises | Search for refund, guarantee, discount and your own list | Machine flags, person decides |
| Wrong tone for this customer | Read it as the customer would | Person |
| Answers the wrong question | Read the enquiry, then the reply | Person |
Which checks can a machine do, and which need you?
A machine should check anything with a right answer, and a person should judge anything that needs taste or context. Prices, names, dates and links are machine work. Whether a reply is kind, sensible and true to your business is yours. Splitting it this way means you read less and catch more.
Machines are good at rules
A check doesn't get bored on the thirtieth email. It will spot that every reply mentions a delivery charge you dropped in the spring, because you told it to look. If you can write the rule as "always" or "never", a machine can check it.
People are good at judgement
A bookkeeper reading a draft to a worried client knows when the tone is wrong for that client this week. No rule captures that. Keep your reading time for the things only you can judge, and let the checks clear the rest first.
Don't let the AI mark its own homework
Asking the same AI "is this right?" can help, but it is not a check. It tends to agree with itself. A second look from a different source, a record, a rule or a person, is what catches the mistake it was sure about. Anthropic's own guidance on building evaluations says it is generally best to grade output with a different model from the one that wrote it, and a record or a person is more independent still.
Common mistakes when checking AI's work
The checks fail most often because of how people use them, not because the checks are weak. These are the habits that let a wrong reply through even when someone is reading.
Reading quickly because it usually looks fine
After a few weeks of good drafts, people start skimming. Researchers call this automation bias: a systematic review in the Journal of the American Medical Informatics Association found that trust, workload and time pressure all make people more likely to over-rely on automated advice. The fix is to keep reading for the four things, not reading less carefully overall.
Trusting it because it shows a source
A reply that says "according to our price list" feels checked. It isn't, until you look. Microsoft's guidance on overreliance on AI notes that the mere presence of sources can make people trust AI output more, even when the sources are wrong. Click the source, or have a check do it.
Checking against memory instead of a record
"That sounds right" is how last year's price gets sent. Every fact check should compare with something written down and current: the price list, the diary, the job sheet. If you can't point to the record, the check is a guess.
Fixing the reply but not the cause
Editing a bad draft takes a minute. Doing it every week for the same reason takes a year of minutes. Each fix should end with one question: which rule, fact or check would have stopped this?
When can you stop reading every one?
You can read fewer once the job has a written record of clean results and the checks catch what you used to catch by eye. Until then, read everything that goes to a customer. Trust is earned by results you can count, not by a few good days.
Keep a simple record
A note of each draft you changed and why is enough. When weeks go by with only small edits, and the reasons are things a check now catches, you have evidence rather than a feeling. If you want a structured way to build that record from day one, see how to trial AI on one job.
Move to spot checks, not to nothing
When you do step back, read a sample every week instead of every reply. A job that was right last month can drift when your prices, hours or services change, so the record keeps going too. Some jobs never move off a person at all: our post on which AI jobs need a person to sign off sets out where that line sits.
Let the checks run where you can see them
Checks only help if their results reach you. The simplest version is a short report each morning: what went out, what was held back and why. That is how Autonomous Agent Operations works, with overnight jobs behind a safety lock and the decisions left for you. For paperwork, Document Automation builds invoices and quotes from the job record itself, so the numbers can't drift from what you agreed.
Questions
08Asked and answered
Not everything, but you should read everything that goes to a customer until the job has a clean record of weeks with only small edits. Internal notes, sorting and labelling can be spot-checked once a week instead. Our post on which AI jobs need a person to sign off sets out where the line sits, job by job.
Read for four things in order: the facts it states, the promises it makes, the name and tone, and anything extra it added. Facts and promises are where the costly mistakes hide, so let automatic checks compare numbers and dates with your records first. To test a job properly before relying on it, see how to trial AI on one job.
It can help as an extra pass, but it is not a real check, because a model tends to agree with what it has just written. Check against something independent instead: your price list, your diary, a written rule or a person. Our safety page explains which actions we always keep behind a human approval.
Look out for wrong facts, promises you don't make, the wrong tone or the wrong person, and additions you never asked for, such as an offer or a link. Wrong facts are the most common, and they usually come from the AI filling a gap in what it was given. We explain where that comes from in why AI makes things up.
Treat it as if you are, because the reply was sent on your behalf, whoever drafted it. For services, the Consumer Rights Act 2015, section 50 can make what you say or write to a customer part of the contract if they rely on it. That is why promises get checked first. For your own situation, ask a legal adviser.
Turn each mistake into a rule in the AI's instructions or a check that runs before anything goes out. One mistake, one new rule, written down the same day. Over a few months the list of checks grows and the reading you do shrinks. We explain the habit in every AI mistake should become a rule.
Yes. Every product we set up includes the checks and approvals for that job, at a fixed price, inside accounts you own. You choose what the AI may send alone and what waits for you, and we write those rules down. Start with how it works, or see every price first.
This is general guidance, not legal or financial advice. For your own situation, speak to a qualified adviser.