How to trial AI on one job before you trust it

A good trial job happens often, has an answer you can check, costs little when it goes wrong, and takes you a known amount of time today. Chasing overdue invoices, replying to booking enquiries or writing up job notes all fit. Your hardest, rarest job does not.

Start where a pass is easy to see. You are testing the set-up as much as the AI, and a simple job shows you both. If you haven't picked a job yet, our post on where to start with AI in a small business helps you choose the first one.

It happens often

A job you do twenty times a week gives you twenty results to judge. A job you do twice a year gives you nothing to measure and nothing to learn from.

It has an answer you can check

An invoice reminder is right or wrong: right customer, right amount, right date. "Write something inspiring for our newsletter" has no pass or fail, so it makes a poor first trial.

A mistake is cheap

Choose a job where a wrong result gets caught before it matters. Drafting replies you approve is cheap to get wrong. Paying suppliers is not.

You know what it costs you now

Time yourself doing the job for a week before the trial. Without that number you can't tell whether the AI saved you anything.

You can trial it without risking customer data

If the job uses customers' personal details, check the plan you are trialling on before you paste anything in. Free and business plans treat your data differently, which we cover in free or paid AI for business. For anything sensitive, the ICO's guidance on data protection impact assessments explains when you need to assess the risk first, including for high-risk processing and innovative technology.

Before the AI sees a single live case, gather a pile of real past ones you already know the right answer to. It turns the trial from "does this look good?" into "did it get these right?", and you can run the same pile again whenever anything changes.

Use real cases from your own records

Take thirty or so past enquiries, invoices or job sheets from the last few months, with names you're allowed to use. Made-up test cases are too tidy. Real ones have the typos, the half-answered questions and the customer who writes in capitals.

Put the awkward ones in on purpose

Include the complaint, the request you can't fulfil, the enquiry with missing details, the regular who expects a discount. Anthropic's guide to building evaluations says tests should mirror the real mix of work and not forget the edge cases, such as missing or ambiguous information. A trial on easy cases only proves the easy cases.

Write the right answer down first

For each case, note what a good result would contain: the amount, the date, the yes or no, the thing it must not say. That note is what you score against, so the scoring is quick and honest.

Prefer more cases to perfect marking

The same guide argues that more test cases with simpler, quicker marking beat a few cases marked in great detail. For a small business, that means a tick, an edit or a cross on thirty cases tells you more than a long essay on five.

Here is a full trial from start to finish. The shop and its tally are an example, but the steps are the ones in this post.

A florist in Norwich gets a steady stream of email enquiries about wedding and funeral flowers. Replying properly takes her most evenings. She decides to trial AI on first replies only: thank them, answer what can be answered from the price guide, and ask for the details she needs to quote.

The set-up

She times a normal week first and writes the pass bar on one page: no wrong prices, no date promised without checking the diary, nothing she would be embarrassed to send. She pulls thirty old enquiries for a test pile, including a funeral order sent at midnight and a bride asking for peonies in November.

The test pile

On the first run, most drafts pass, but the AI promises peonies in November and quotes a delivery charge for a village outside her area. Both come from gaps: the price guide had no seasonal list and no delivery map. She adds both, reruns the pile, and those two failures turn into passes.

Four weeks live, with approval

For four weeks every draft waits for her. She ticks, edits or rejects each one and notes why. The edits cluster around funeral enquiries, where the AI's cheerful opening is wrong. She writes one rule for tone on sympathy orders and keeps scoring.

The decision

On the end date, the tally clears the bar she set. The job is handed over with her still approving every reply, and the sympathy rule stays. She now reads drafts in the time it used to take to write a few of them, and she knows that from a count, not a feeling.

A trial is only as good as its rules. These are the slips that make the result meaningless, in either direction.

Judging it on the demo

A demo is the best case, chosen by someone who wants you to buy. Your trial is the normal case, chosen by you. Only the second one tells you anything about your business.

Changing the rules halfway without saying so

Improving the instructions mid-trial is fine, and usually the point. But mark the date, and rerun the test pile, so you know whether the later results came from the change or from easier weeks.

Trialling three tools at once

Comparing tools sounds thorough and usually means none of them is set up properly. Trial one, with a proper set-up. If it fails for reasons that look like the tool, try another on the same test pile and bar. Our post on which AI model a small business should use helps you pick the first.

Counting the good ones and forgetting the bad

One wrong price can cost more than twenty good replies save. Weigh failures by what they would have cost, not just how many there were. That is why facts sit on "every one, no exceptions" in the bar.

When the trial ends, compare the tally with the bar you wrote and do one of three things: hand the job over with sign-off, fix the missing rule and run a short second trial, or stop. All three are good outcomes, because each one is a decision made on evidence.

It passed: hand it over, with sign-off

Let the AI do the job for real, with a person still approving anything that goes to a customer. Keep the tally going. A job that passed in October can slip when your prices change in January. Our post on which AI jobs need a person to sign off sets out when that approval can safely loosen.

It nearly passed: find the missing rule

Look at the failures together. They usually share a cause: a fact it didn't have, a rule nobody wrote down, a kind of customer it didn't expect. Add the rule, then run a shorter trial on the same bar. Each failure you fix this way should stay fixed, as in every AI mistake should become a rule.

It failed: stop, and keep the notes

Some jobs aren't ready for AI yet, or need a process fixed by hand first. Stopping is cheap. The notes tell you what would have to change, so a later trial starts ahead.

Rerun the test pile when anything changes

The model underneath can change without you doing anything. The National Cyber Security Centre warns in Exercise caution when building off LLMs that providers can update models in ways that break what worked before. Keep your test pile and rerun it after a model update, a new price list or a new tool. It takes an hour and tells you whether the job still passes. Our post on keeping up with AI without chasing launches covers when an update is worth acting on.

Make it part of how you run things

Once one job has been through a trial, the next one is quicker: you have a template for the pass bar and a habit of scoring. If you want a wider frame, NIST's AI Risk Management Framework is a voluntary framework built around four functions, govern, map, measure and manage, and the UK government's AI Management Essentials is a self-assessment tool it consulted on with start-ups and small firms in mind. Neither is required; both are useful checklists.