Harness engineering
Every AI mistake should become a rule
The AI told a customer you open on Sundays. You don't. You corrected the reply, apologised, and moved on. Three weeks later it said it again.
What does turning a mistake into a rule mean?
It means every mistake changes the set-up, as well as the answer. The wrong reply gets fixed, and then something is added so the AI is told, or stopped, next time. Over months, the list of things it can get wrong gets shorter.
Engineers call this a ratchet. Each turn holds. A mistake that has been turned into a rule doesn't quietly come back after a busy week, because the rule is written down where the AI reads it every time.
Fix the cause, not only the answer
Correcting the Sunday reply fixes one customer. Asking why the AI thought you opened on Sundays fixes every customer after. Usually the answer is plain: the opening hours were never in its brief, or an old page still said Sundays.
A rule or a check
A rule is a line in the AI's written instructions: "Our opening hours are Monday to Saturday, 8 till 6. Never say we are open on Sunday." A check is something that catches the mistake whether or not the AI follows the rule: a person reviewing any reply that mentions opening times, or software that flags it.
Why a check beats a louder rule
If the AI breaks a rule twice, writing the rule again in capitals rarely helps. The second time, add a check. A rule makes the mistake less likely; a check means it can't reach a customer unnoticed. Anthropic says the same of its own coding tool: written instructions are treated as context, not enforced settings, and to block an action whatever the AI decides you add a hook, a small piece of code that always runs, in its guide to hooks.
An old idea in a new place
Safety teams have worked this way for decades. The Health and Safety Executive's workbook on investigating accidents and incidents is built on finding what failed and acting on it, because learning lessons and taking action can reduce or prevent the next accident. An AI mistake log is the same habit, applied to a much smaller kind of incident.
The AI companies say the same
Anthropic's page on how Claude remembers your project tells users to add to the AI's written instructions when it makes the same mistake a second time, or when you find yourself typing the same correction you typed last session. That is the ratchet in one line, from the people who make the model.
- 01Mistakeone line: what happened, when, who saw it
- 02Causemissing fact, wrong source, vague rule
- 03Rule, fact or checkadded where the AI reads it every time
- 04Testreplay the job that tripped it up
- 05Logdated, so the next person can search it
How to run the ratchet in a small business
- 01
Write the mistake down in one line
What happened, when, and who saw it. "Tuesday: reply to a new enquiry said we open Sundays." A notebook or a shared document is enough. The line takes thirty seconds and is worth more than any memory of it a month later.
- 02
Find why it happened
Was the fact missing from the AI's instructions? Was it wrong in a document the AI reads? Was the rule there but vague? Most mistakes have one of these three causes. Invented facts have a few more, covered in why AI makes things up.
- 03
Decide: rule, fact or check
A missing fact goes into the brief. A vague rule gets rewritten. A mistake that has already happened twice, or one that costs money, gets a check. The table below shows which fix suits which cause.
- 04
Add it where the AI will see it
Edit the saved instructions or the source document itself, not a single conversation. A fix typed into one chat is gone when the chat ends. How to keep that brief clear as it grows is in how to write instructions an AI will follow.
- 05
Test it against the original mistake
Give the AI the same enquiry that tripped it up. If the answer is now right, the ratchet has turned. If not, the rule needs rewriting, or the mistake needs a check instead.
- 06
Keep the tricky jobs as a test set
Save every enquiry that once caught the AI out. After any change to the brief, the tools or the model, run the whole set again. It takes minutes and tells you straight away if an old mistake has come back.
Which fix for which mistake
| What went wrong | Example | The fix | Where it goes |
|---|---|---|---|
| A fact was missing | A café's AI said it had oat milk; it had run out for good in March | Add the fact | The brief, or the menu document it reads |
| A source document was wrong | An old price list was still in the shared folder | Fix or remove the document | The folder the AI reads from |
| A rule was vague | "Be brief" gave a plumber's customers one-line replies | Rewrite with a number and a reason | The brief |
| A clear rule was broken twice | A bookkeeper's AI kept adding VAT to exempt items | Add a check | A review step, or software that flags it |
| The mistake was expensive | A refund went to the wrong customer | Take the power away | The AI's access settings |
| The same problem survived two fixes | Booking replies stay muddled however the brief is worded | Stop and rethink the job | The process itself, on paper |
A worked example: three months of a café's mistake log
Here is what the ratchet looks like in practice, for a café that uses AI to answer messages about bookings, allergens and opening times. The café is made up; the pattern is typical.
Month one: mostly missing facts
Six entries. It said the café opened on Sundays. It offered a table for twelve when the biggest booking is eight. It said a cake was nut-free when the bakery couldn't promise that. Each one became a line in the brief or a fix to the menu document, and the allergen question became a check: every reply that mentions allergens waits for a person.
Month two: fewer, and different
Three entries, all new kinds of mistake. None of the month-one mistakes came back, because the test set caught one that nearly did after a menu change. One rule was rewritten because it clashed with another.
Month three: quiet
One entry, about a new seasonal menu the AI hadn't been given. The fix took two minutes. The log now holds ten lines, and the owner reads it once a week with a coffee.
What it cost in time
Example figures, not a promise: around five minutes per mistake to log, find the cause and add the fix, so under an hour across the three months. Compare that with apologising to the same kind of customer for the same mistake every few weeks.
When should a mistake become a check?
Make it a check when the mistake has happened twice, when it would cost money or trust, or when it is easy to spot by rule. A check catches the mistake whether or not the AI follows its instructions, so it is the right fix for anything that must never reach a customer.
A person, at first
The simplest check is you. Any reply that mentions a price, an allergen or a refund waits for your approval. How to check AI's work before a customer sees it sets out a quick way to review drafts without reading every word.
Simple automatic checks
Many logged mistakes can be caught by a plain rule in software: does the reply contain a price that isn't on the list, the word Sunday, a phone number that isn't yours? Developers call a set of these tests evals. OpenAI describes them as tests that check the AI's output meets criteria you set, in its guide to evals.
Test the edge cases, not just the easy ones
Anthropic's guide to building evaluations advises testing against your real mix of work and remembering the edge cases: the empty message, the very long one, the question even a person would find hard. Your mistake log is the best source of those, because each entry is a real one.
Re-run them when anything changes
A new model, a new price list or a new tool can bring an old mistake back. Run the test set after every change. The UK government's own AI Playbook asks for the same: testing and monitoring in place to catch problems such as made-up answers once a tool is live. That habit is also the calm way to handle new AI releases, as keeping up with AI without chasing launches explains.
When a new rule isn't the answer
The ratchet is a habit, and like any habit it can be overdone. Four signs you need something other than another rule.
The rule list keeps growing
Forty rules for one job usually means the job is really two jobs, or the instructions need rewriting from scratch. Read them together, merge the ones that overlap and delete the ones that no longer apply. Anthropic's own advice for its coding tool is to keep each instructions file under 200 lines, because longer files are followed less closely.
The same mistake survives two fixes
We have a house rule for this: a problem that survives two fixes is being looked at the wrong way. Stop adding rules. Step back and ask whether the AI should be doing this job at all, or whether the process behind it is muddled.
The mistake would be expensive
Some errors are too costly to learn from even once: a refund to the wrong customer, a deleted file. Those need a limit on what the AI can do, decided before it starts, rather than a rule written after. Give AI only the tools it needs covers how to set those limits.
The mistake is really a judgement call
If the right answer depends on reading a person, such as whether to waive a fee for a regular, no rule will get it right every time. That job keeps a person on it. Which AI jobs still need a person to sign off helps you sort them.
Keep the log useful for years
A mistake log is only worth keeping if you can find things in it. A few habits keep it short, searchable and honest.
Write symptom, cause and fix
Three short parts per entry, with a date: what you saw, why it happened, what changed. Search for the symptom later and the fix is right there.
Keep it next to the brief
Store the log in the same folder as the AI's instructions, so whoever edits one sees the other. Documents that drift apart cause the next mistake. Our Living Documentation adds checks that fail when a document falls behind the system it describes.
Watch it when you are not looking
Some mistakes only show up when nobody is checking, such as a booking form that quietly stops working. A scheduled test catches those. Synthetic Monitoring tests your key customer journeys every 2 hours.
Questions
08Asked and answered
Fix the answer, then find why it happened and add a rule to the AI's instructions or a check that catches it next time. Write one line in a mistake log with the date, the cause and the fix, then replay the original job to prove the fix works. The idea sits inside the wider set-up described in what harness engineering is.
A ratchet is a habit where every AI mistake adds a rule or a check, so the set-up only ever gets tighter and the same error doesn't come back. Each turn holds because the fix is written where the AI reads it every time, or enforced by a check it can't skip. We run it on our own systems, as our proof page shows.
Write the rules in the AI's saved standing instructions for that job, so it reads them every time rather than once in one chat. Facts belong in the source document the AI reads, such as the price list, so there is only one copy to keep right. How to write that brief is covered in how to write instructions an AI will follow.
Add a check before anything leaves: a person reviewing replies at first, then simple automatic checks for the mistakes you have already logged. Prices, allergens, refunds and complaints are the usual first candidates for a person's approval. The full method is in how to check AI's work before a customer sees it.
More than a page of rules for one job is usually too many, because long instructions are followed less closely. When the list grows past that, merge rules that overlap, delete ones that no longer apply, and move rarely needed detail into a separate document. Anthropic suggests keeping each instructions file for its coding tool under 200 lines, in how Claude remembers your project.
The AI will make fewer of the mistakes you have logged, but new ones can always appear, especially after a change of model, menu or price list. Keep the log going, re-run your test set after every change and keep a person on the jobs that matter. Our safety page sets out what AI never does without you.
Yes, with checks that fail when a document falls behind the system it describes, which stops old facts creeping back into an AI's answers. Without them, an out-of-date page is one of the commonest causes of a repeat mistake. That is what Living Documentation sets up, at a fixed price.
This is general guidance, not legal or financial advice. For your own situation, speak to a qualified adviser.