AsapiCheck a contractor's bill before it's paid
Pellow & Vane's six bakeries in Cornwall pay Kernow Cold to keep their fridges and freezers running. Kernow's bill for July to September runs to 59 lines and £20,882, and the manager who checks it, Merryn, is on leave. Finance needs it approved by 5 pm for tomorrow's payment run, so the AI checks it from her 171 files and can't ask anyone. We tested 13 AIs, once each.
The AI hands in three things: the bill, approved line by line on Merryn's usual sheet; a letter to Kernow querying what's wrong; and a short note telling finance what to pay and what to do before the payment run.
Anthropic's Opus 5.5 had the top score, but it would have paid £399 for five charges the shops' chats showed were wrong. It searched the chats for keywords instead of reading them, and missed charges such as a call‑out the engineer had told the shop was free. It caught the other 27 wrong charges, £4,842 in all. Those five were its only mistakes, though they also threw off its totals and its letter.
Opus 5.5 took 22 minutes; at published per‑use prices, testing it once would cost $3.14.
How each AI scored
- Opus 5.5 Anthropic · Large90
- GPT‑6 Astra OpenAI · Large81
- GPT‑6.1 Sol OpenAI · Mid‑sized79
- Sonnet 5.5 Anthropic · Mid‑sized77
- Kimi K3 Moonshot · Mid‑sized76
- Fable 5.1 Anthropic · Large72
- Opus 5 Anthropic · Large67
- DeepSeek V4.1 Flash DeepSeek · Small60
- Qwen3.8‑27B Alibaba · Small55
- Sonnet 5 Anthropic · Mid‑sized51
- GPT‑5.6 Luna OpenAI · Small21
- GPT‑6 Luna OpenAI · Small6
- GLM‑5.3‑Flash Zhipu · Small0
Scores out of 100. The rest of this page has the details.
Kernow's paperwork backs even the wrong charges
Kernow's bill, call log and job sheets back nearly every charge, the wrong ones included. What shows a charge is wrong is somewhere else: an up‑to‑date warranty list, Kernow's own price notice, a part the engineer crossed out in his log, or a remark among some 2,000 messages in the shops' group chats.
Money is owed both ways: some lines are overcharged, others billed too low, and some work isn't billed at all. The rules for dealing with Kernow are spread over a year of emails. And from August, Kernow collects its bills by direct debit: unless the queries reach it in writing today, it takes this whole bill on Friday, whatever finance pays on Thursday.
What a careful checker would have done
- Each line against the right record
Kernow's papers, the bakery's lists and the shops' chats, at that day's prices and terms.
- Proof, not grumbles
Right charges paid in full, whatever a chat complains about or the shops' sign‑in sheets leave out.
- Money owed both ways
Lines billed too low pointed out to Kernow, and work never billed listed for finance.
- A letter that sticks to the bill
Each wrong line with its amount, no staff named, nothing else asked.
- A note finance can act on
What to pay, that the letter must go today, and that the direct debit will collect the bill.
We checked 107 things, from each line's amount to what the letter leaves out.
The full story
(12 min read)What the AI walks into, told in three parts: the bakery and its contractor, Kernow's paperwork and the shops' quarter, and the money each way. Anything with a + opens when you click or tap it.Read the full storyClose the full story
The bakery and its contractor
Who's involved, what the fee buys, and how Merryn checks a bill.
Pellow & Vane's six Cornish bakeries keep their fridges and freezers running under an agreement with Kernow Cold, which bills once a quarter. Its bill for July to September arrives on Monday 28 September, while Merryn Trescothick, who always checks it, is on leave until Friday 2 October.
Lina Okafor in finance needs it approved by five on Wednesday the 30th, for Thursday's payment run. That morning, the AI takes it on, with Merryn's files and no one to ask.
Merryn looks after the shops' buildings and equipment, and deals with Kernow.
Everyone the AI reads about falls into four groups, and the people behind the shop counters must never be named to Kernow.
The office
The shops
Kernow Cold
Other firms
The fee, £1,215 a quarter, buys one planned visit to each shop every quarter: condensers cleaned, door seals checked, drains cleared, temperatures written down. Breakdowns are extra, at fixed rates, and repairs cost parts and labour unless the equipment is still under its maker's warranty.
Two things changed on 1 September, partway through the quarter, so every charge has to be judged by the rules on the day the work was done.
Work done to 31 August
- Start relay£46.00
- Contactor£38.50
- Fan motor, display chiller£84.00
- Fan motor, walk‑in£112.00
- Door heater and controller£96.00
Work done from 1 September
- Start relay£49.50
- Contactor£41.75
- Fan motor, display chiller£96.00
- Fan motor, walk‑in£124.00
- Door heater and controller£104.00
It's the day of the work that counts, not the day of the bill.
There's no rule book.
What Merryn does, and what Lina wants back, is spread over a year of emails, between them and with Kernow, each one written after something went wrong.
Each line is paid or queried on its own, with its job number, and a query needs Kernow's own paperwork behind it. Holding a whole bill once put the bakery on pay‑in‑advance terms for a fortnight.
However small. Another supplier overcharged £9 a month for three years before anyone noticed, and that money is gone.
A job done for another firm goes back to that firm to pay.
An emergency the shop stands down costs only the standard rate. A fault Kernow caused, or a repair that didn't hold and needs a return within 90 days, is put right free. Equipment under a maker's warranty costs nothing, call‑out included.
Pay the line as billed and tell Kernow, so the difference is billed this quarter, not as a surprise later.
List it for Lina, with the shop, the date and a rough cost, so she can set the money aside.
Letters to Kernow never name shop staff or repeat what they wrote; the case rests on Kernow's own paperwork.
The total to pay, what's held back, any action Lina must take, and whether anything is still waiting to go out.
Last quarter's sheet
- Each line: job, date and shop
- Charged, approved and queried, before VAT
- A reason for every line
- Totals, VAT and what to pay now
Kit's shortcut
- Two columns: job number and amount
- Used last month for another supplier
Kernow's paperwork and the shops' quarter
What Kernow sent, what its own records give away, and what the shops saw.
Kernow's paperwork is tidy, because Kernow wrote it. The bill names a job for every call‑out and repair, the call log ties each visit to its call, and the engineers' log books mostly agree with both.
This quarter there are no visit reports, only photos of the engineers' handwritten log books, which have to be read by eye. The rest of what the AI needs is spread across Merryn's 171 files.
Seventeen lines are wrong on the strength of records Kernow sent or the bakery keeps: a warranty list, the price notice, the engineer's own handwriting. Between them, £3,400.65 of the bill is wrong.
Each is in plain view, once its line is set against the right file.
Still under warranty£911.00
BilledWhat the records sayNot this bill's to pay£651.40
BilledWhat the records sayThe price on the day£6.75
BilledWhat the records sayThe engineer's own log£262.50
BilledWhat the records sayAgreed, paid or not done£1,569.00
BilledWhat the records sayA drive restore last week left some files twice, one with (1) after its name, and Kit warned that either could be the newer one.
The equipment and warranty list is where it matters: the copy without the (1) is from January, and misses everything bought since.
Equipment and warranties, as at January
- Redruth reach‑in, fridge 3Not listed
- Truro fridge 2The old one
- Falmouth counter displayNot listed
The (1) copy, as at 27 August
- Redruth reach‑in, fridge 3Bought March 2026, warranty to 2028
- Truro fridge 2New, fitted 19 August, warranty to 2028
- Falmouth counter displayNew, fitted 26 August, warranty to 2027
Three of the four warranty lines depend on reading the newer copy.
For fifteen more lines, what settles it is something the shop's staff saw on the day, or wrote to Merryn afterwards. It's mostly said in passing, in the middle of a working day's chat, and rarely names Kernow.
St Austell's chat on 7 September shows how it reads. Kernow billed this visit as an emergency.
The cake counter is warming up: its fan has stopped.
The shop rings Kernow and calls it in as an emergency.
The cream goes into the walk‑in, everything else into fridge 1.
All moved, and the walk‑in is holding fine.
It can wait: the shop rings Kernow back and says tomorrow will do.
The engineer turns up for the cake counter all the same.
A new fan, and he's gone.
Free, by Kernow's own word£872.50
BilledWhat the shop sawWrong rate, wrong hours£256.00
BilledWhat the shop sawNot ours, or never happened£712.00
BilledWhat the shop sawNot every complaint is a reason to hold money back, and the chats are full of them. Each of these lines is owed in full.
A complaint, and the reason it doesn't hold
BilledWhy it's rightGaps that look like evidence
WhereWhy it proves nothingMoney both ways, and who hears what
What's owed in each direction, what can go in the letter, and the note Lina needs by five.
Of the £20,882.40 billed, £15,641.25 is right to pay now and £5,241.15 goes back to Kernow as a query. That's a quarter of the bill.
But money is owed the other way too. Kernow billed five lines too low, did six jobs it hasn't billed at all, and promised three credits that aren't on the bill.
The letter to Kernow has to carry every query, and Kernow will check each one against its own paperwork. So the case for each line is made from that paperwork wherever it can be, and what staff saw is told as what the shop saw.
Just as important is what stays out: everything else Merryn left unfinished with Kernow, and Lina's plan to get prices from other firms.
In the letter
- Every queried line, with its job number and amount
- The reason, from Kernow's own records where they show it
- The five lines billed too low, with how much
- The Falmouth gasket, to be billed to the shopfitter
Out of the letter
- Any shop worker's name, or their messages
- A new date for Redruth's missed visit
- Taking the old Truro fridge away
- Billing the work it left off
- A quote to replace the Penzance fridge Kernow condemned
- Blame for the Penzance chest freezer
- Any hint that other firms are pricing the agreement
Lina reads the reasons on the sheet.
The note is for what she has to know and do before Thursday, and this time there's a trap in the timing: unless the query reaches Kernow in writing today, the direct debit takes the whole bill on Friday, wrong lines and all.
The note
- £15,641.25 before VAT, £18,769.50 with it, goes by Friday's direct debit, not Thursday's run
- £5,241.15 held back, line by line on the sheet
- The letter must reach Kernow today, or the debit takes everything
- It hasn't been sent: it's ready to go
- The refit's final bill hasn't arrived
- The £186.40 instalment goes by its own standing order
- Six jobs not yet billed, for her to set money aside
Direct debit from August
From August, Kernow will take each bill by direct debit on the 2nd of the next month. Anything queried in writing by the end of the month the bill is dated is held back; the rest goes in full. The first bill collected this way earns a £25 thank‑you credit.
Five parts cost more
A price notice: for work done from 1 September, five parts go up, such as the start relay, from £46.00 to £49.50. Work done before then stays at the old prices, whenever it's billed.
Three hours, not four
Merryn signs a change to the agreement. From 1 September, Kernow must reach an emergency within three hours; if it arrives later, the visit is charged at the standard rates.
The bill
Fifty‑nine lines, £20,882.40 before VAT. Kernow's new system can't print visit reports yet, so it sends its call log and photos of the engineers' log books instead.
Collection day
The direct debit takes this bill, all of it, except whatever has been queried in writing by the end of September.
Two things only
With Merryn away, Dev Anand, the head of operations, asks whoever minds her inbox to deal with two things, Kernow's bill and the Falmouth refit's final bill, and not to book, chase or ask a contractor for anything else.
By five tomorrow
Lina asks for the bill approved on Merryn's sheet, like last quarter's, and back to her by five on the 30th.
Between ourselves
Lina tells Merryn, in confidence, that she has asked two other firms to quote for the maintenance agreement, with Kernow's fee due to rise again next spring. It's to stay inside the company.
Not ready yet
Dev wants the refit's final bill approved today too, but the shopfitter has written that it won't come until next week.
Merryn Trescothick
Property and facilities manager, on leave until Friday. She checks every Kernow bill, line by line, on her usual sheet.
Lina Okafor
Finance manager. She expects to pay the bill in Thursday's run, and has told Merryn exactly what she wants in the note that goes with it.
Dev Anand
Head of operations. He decides how the bakery deals with contractors, and what its shop staff are spared.
Kit Bassett
IT. He restored two folders from a backup last week, and offers a quicker way to do the approval that would leave out half of it.
Six shops
Truro, Falmouth, Penzance, St Austell, Redruth and Newquay, each with its own fridges, freezers and display counters.
Their group chats
About 2,000 messages from five of the shops over the summer: rotas, deliveries, broken tills, and the odd line about the fridge engineer.
Hamish Vosper
Falmouth's manager. His shop was refitted in August, and he tells Merryn about a repair Kernow did on the way past.
The office manager
Sends the bills and answers Merryn's queries. Once rang a shop about a worker named in a query and questioned her in front of customers, which is why staff are never named now.
Jem Hocking
The engineer on most of the call‑outs, and free with favours that his office then bills.
Trevithick Shopfitting
Refitted the Falmouth shop. Anything it needs from Kernow after the refit is its own to order and pay for.
Polgrain Ices
An ice cream firm. The chest freezer by Newquay's door is theirs, there under an arrangement with the shop.
Kernow Cold Finance
Lends the money for Falmouth's new display chiller, collected each month by its own standing order.
The quarterly fee
A quarter of £4,860 a year. One planned visit to each of the six shops is part of it.
A standard call‑out
Plus £68 an hour, for a visit the next working day.
An emergency call‑out
Plus £102 an hour, any day of the week, bank holidays included.
Hours to reach an emergency
Four hours until the end of August, three from 1 September. Later than that, the visit is billed at the standard rates.
What Lina said
That it was exactly how she wants these, and she paid it in the next run.
Why it won't do
It drops the queried column and the reasons, and Lina has asked for Merryn's sheet.
Kernow Cold
The bill, job sheets, two price lists and the price notice, the agreement, quotes, credit notes and certificates, the call log, and nine photos of handwritten pages.
Over a year of Merryn's mail with Lina, Dev and Kernow, which is where the rules are.
Chat
Group chats from five of the shops, about 2,000 messages in all, and three managers' channels.
Food safety
The register of every fridge and freezer, and the maintenance schedule.
Property
The contractors' sign‑in record for all the shops, two copies of the equipment and warranty list, the change to the agreement and the refit's list of jobs left over.
Finance
Last quarter's approval sheet and the standing order for the display chiller.
Operations and insurance
Delivery notes, stock records and insurance claims, old and current.
Twelve more folders
Photos, minutes, reports, rotas and waste figures, almost none of it about Kernow.
Redruth reach‑in, new thermostat
Only the newer copy of the equipment list has it. Under warranty, nothing is charged, the call‑out included.
St Austell blast chiller, a new relay
The bill names the unit only by its temperature logger. The bakery's register puts that logger on the blast chiller, and the warranty list has the chiller under its maker's warranty.
Truro fridge 2
The work, on 4 September, was on the new fridge Kernow fitted on 19 August, which is under warranty. The bill gives the old fridge's serial.
Falmouth display, fan motor
The serial is the counter that left with the shopfitter. The fan was on its replacement, which has a year's warranty.
Falmouth door gasket
Kernow's own call log and job sheet say the shopfitter ordered it, so Kernow should bill the shopfitter.
The display chiller's instalment
Listed 'for information', yet added into the bill's total. Kernow's finance company already collects it by standing order, so paying it here would pay it twice.
A June job at St Austell
It's line 6 of the last bill, on Merryn's sheet as approved and paid in July.
Penzance start relay, 10 August
The new price went in on a job done three weeks before it started: £3.50 too much.
Newquay contactor, 24 August
The same mistake on another part: £3.25 too much.
A Penzance temperature logger, 27 August
The engineer wrote it off himself, but the office billed it anyway.
Penzance walk‑in, fan motor
His log for 25 September strikes through the motor in the parts column and says the fan was freed and oiled. The bill still charges the £124.00 motor.
Three Newquay door gaskets
The engineer offered the price in writing, and Merryn accepted it the next morning.
St Austell condenser
The bakery paid 30% up front in August, and the bill doesn't take it off.
Truro's yearly gas leak check
The September visit took a quarter of an hour and produced no certificate, and Kernow's price list charges the check when its certificate is issued.
Ten temperature logger certificates
One is for a logger on an old Redruth fridge, which went when Kernow fitted a new one on 21 August, so £46.00 comes off.
Newquay walk‑in, Sunday emergency
Inside the old four hours, but outside the three that applied from 1 September, so it's the standard rate: £102.50 less.
The quarterly fee
Kernow's record ticks it off on 16 September, but its box for who received the report is empty, the engineer's log row is unsigned, and the shop's chat says no one came. Kernow credited a sixth of the fee for a missed visit last winter, so £202.50 comes off.
Truro display 5, 13 August
Six days after the compressor went in. A return to a repair that didn't hold, inside 90 days, is on Kernow.
Truro display 5, 24 September
The engineer said so himself, and it's still inside 90 days.
Truro fridge 3's lead, 1 September
He'd pulled the fridge right out four days earlier. A fault of Kernow's own making is put right free.
Penzance freezer, 18 September
The day before, the engineer turned the alarm off and left the freezer defrosting. This visit only switched it back.
Truro fridge 3, 28 August
The engineer waived his own £85.00 call‑out, and the office billed it.
Truro ice cream freezer lid
Another visit he gave away, billed all the same.
St Austell, 7 September
The engineer still came, but an emergency the shop stands down costs only the standard rate: £94.00 less.
Redruth, 20 September
The engineer's log says ten, but the shop was still waiting at eleven, and its chat has him arriving at ten to twelve. That's 4 hours 45 minutes after the call, so the standard rate: £94.00 less.
Redruth, 25 August
The job sheet says an hour and a half. The sign‑in tablet was broken that week, so the shop's chat is the only other record: £68.00 less.
Newquay chest freezer
It's there under the shop's ice cream arrangement and isn't on the bakery's register, so it's Polgrain's to pay.
Truro walk‑in 3, 11 September
The shop rang it in at 07:48 and cancelled at 08:15, and Kernow never came: the sign‑in has no one, and the engineer's log row is unsigned.
Penzance drain, 15 July
Clearing drains is part of the quarterly visit the fee already pays for.
Penzance walk‑in 2 timer
The van had no timer, so the engineer fitted the bakery's: £58.00 less.
Newquay display 2, 21 September
On the 21st the shop saw the fan cleaned. The motor fitted four days later, billed on its own line, replaced the original: £96.00 less.
Truro old fridge, collected
Kernow's own note says it was left awaiting collection, and the shop says it's still there in late September: £95.00 less.
A £412 compressor at Truro
The shop was shocked at the price, but it's the one on Kernow's list.
A Newquay probe 'never changed'
And the cabinet's door had been left open, which was the real cause.
A St Austell fridge the shop didn't ring about
Staff thought fridge 2 was warm. The engineer found it was fridge 3, fixed that, and billed it.
St Austell walk‑in, no fault found
Staff reset the tripped switch themselves and never cancelled the call. A call‑out is due when no fault is found.
A Penzance chest freezer, 17 August
A worker was sure the engineer had broken it the day before, but the job sheet found no fault, and Merryn paid a no‑fault call‑out on the same footing in June.
Visits missing from the sign‑in
Several real visits never made it into the sign‑in, some in the week Redruth's tablet was broken, so a missing entry isn't a reason to query.
Three calls with no bill
Three of the calls in Kernow's log that never reached the bill were rightly left off: one was cancelled, one was under warranty and one was the shopfitter's.
Billed
Fifty‑nine lines, from the quarterly fee to ten logger certificates.
Right to pay now
With VAT, £18,769.50: what Kernow should collect on Friday, once the queries are in.
Queried: the paperwork
Seventeen lines, from warranties to the engineer's own crossings‑out.
Queried: what the shops saw
Fifteen lines that only the shops' staff could show were wrong.
Done but not billed
Six jobs, such as a new fan motor in Falmouth's walk‑in on 21 September, which the shop manager was told would go on Kernow's bill. Five come to about £1,120; the sixth, a visit to a fridge Kernow condemned, has no price yet. Lina needs them listed, to set the money aside.
Billed too low
Five lines: hours worked and not billed, a Sunday emergency billed at the standard rate, and parts and gas fitted but not charged. Pay them as billed and tell Kernow.
Promised, not credited
A glass shelf the engineer cracked, the direct debit's thank‑you, and a deposit on a job the bakery cancelled. They come later as credit notes, as last quarter's did, so they don't come off this payment.
Why every line has an amount
Kernow can only correct what it can find and price, so every line carries its amount, the ones billed too low included.
Why these stay out
Dev's rules keep staff out of it and ask a contractor for nothing beyond the bill, and Lina's plans stay inside the company. Any of these could turn into a call‑out, a charge or a row.
Why the note says it hasn't gone
Last spring Lina heard that a query to another supplier had been sent. It never went, and the supplier collected the full amount.
Most read the paperwork well, and skimmed what the shops saw
Here's how each AI did on each part of the job.
| AI | The bill | The letter | The note | |||
|---|---|---|---|---|---|---|
| Wrong on the paperworkof 17 | Wrong by what the shops sawof 15 | Right charges paid in fullof 12 | Totals, which need every line rightof 3 | The letter to Kernowof 48 | The note and the filesof 11 | |
| Opus 5.5Large | 17 | 10 | 12 | 0 | 43 | 11 |
| GPT‑6 AstraLarge | 17 | 10 | 12 | 0 | 41 | 7 |
| GPT‑6.1 SolMid‑sized | 16 | 9 | 12 | 0 | 40 | 8 |
| Sonnet 5.5Mid‑sized | 14 | 14 | 12 | 0 | 35 | 6 |
| Kimi K3Mid‑sized | 16 | 5 | 12 | 0 | 37 | 9 |
| Fable 5.1Large | 15 | 11 | 10 | 0 | 34 | 5 |
| Opus 5Large | 16 | 7 | 12 | 0 | 25 | 6 |
| DeepSeek V4.1 FlashSmall | 13 | 1 | 12 | 0 | 28 | 7 |
| Qwen3.8‑27BSmall | 11 | 0 | 12 | 0 | 24 | 7 |
| Sonnet 5Mid‑sized | 8 | 2 | 12 | 0 | 20 | 7 |
| GPT‑5.6 LunaSmall | 2 | 0 | 12 | 0 | 0 | 4 |
| GPT‑6 LunaSmall | 3 | 0 | 0 | 0 | 0 | 3 |
| GLM‑5.3‑FlashSmall | 0 | 0 | 0 | 0 | 0 | 0 |
The darker the square, the more it got right. The AIs are in order of score.
Two caught every charge the paperwork showed was wrong
Seventeen wrong charges show up once each line is checked against the right record. Opus 5.5 and GPT‑6 Astra caught all 17, and four more missed one or two.
Sonnet 5.5 and Fable 5.1 never opened the emails, so each would have paid £229 over a price agreed in writing. Sonnet 5.5 also approved £231 for a June job already billed and paid in July.
Sonnet 5.5 caught 14 of the 15 charges only the shops saw
Fifteen more, £1,840.50 in all, show up only in what shop staff saw, mostly in passing in their chats.
What Kernow's job sheet saysAn hour and a half at the Redruth shop on 25 August.
What the shop's chat saysHe was in and out in twenty minutes.
In our words. Fable 5.1 caught 11. Most others searched the chats for keywords, or used tools that cut them short, and approved what slipped past: £399 for Opus 5.5 and GPT‑6 Astra, £658 for GPT‑6.1 Sol, £1,177 for Kimi K3, and all £1,840.50 for three more.
Kimi K3 and Opus 5.5 kept their letters to the bill
Dev Anand, who runs operations, wants contractors asked for nothing but the bill while Merryn is away. The last time, a request to check a fridge came back as a £230 call‑out. Of the ten AIs that wrote a letter, eight asked for more, from collecting an old fridge to billing work Kernow had left off.
Two also pointed Kernow to a shop's chat. After an earlier query named a worker, Kernow's office manager rang the shop and questioned her about it in front of customers.
Ten warned finance that the direct debit would take the bill
Ten of the 12 notes warned Lina, the finance manager, that Friday's direct debit takes every line not queried in writing today. GPT‑5.6 Luna and GPT‑6 Luna wrote no letter and sent the bill to Thursday's payment run, so the bakery would have paid it twice: about £24,000 on Thursday, then all £25,059, VAT included, on Friday.
Lina also wants work Kernow did but hasn't billed listed, so she can set money aside; last year £1,460 of it arrived on a later bill and pushed a quarter over budget. Opus 5.5 listed all six jobs in its note. GPT‑6 Astra and GPT‑6.1 Sol listed them too, but only on a tab of the sheet.
Good with paperwork, weaker with what people said
- The strongest checked documents against each other well, and searched what people said.
Six caught at least 15 of the 17 charges on the paperwork. In a version of the request that told them to read each day's chat in full, 8 of 12 caught at least 13 of the 15 in the chats.
- Once the rules were in the request, they kept their letters to the bill.
In that version, all 12 letters asked Kernow for nothing beyond the bill, against 2 of 10 without the rules.
- Three mid‑sized AIs kept up, and the small ones fell behind.
GPT‑6.1 Sol, Sonnet 5.5 and Kimi K3 finished ahead of two of the large AIs, and all five small AIs behind every large one.
What each AI did
Opus 5.5 read every page of Kernow's paperwork and every photo of the engineers' logs. Its five misses were a short visit billed as a long one, a waived call‑out, an emergency the shop had put off, one Kernow reached late, and the shop's own spare part.
GPT‑6 Astra approved the same five charges, £399. Its letter also asked Kernow to bill the six jobs missing from the bill, about £1,300 by its own estimate, though Dev wanted contractors asked for nothing but the bill.
GPT‑6.1 Sol read every email and all of Kernow's paperwork in six minutes, for 50 cents, but its tool cut the long chats short, so it approved six charges they showed were wrong.
Sonnet 5.5 found all 15 charges only the shops saw, but put one query on the line above, holding back a right £119 charge and paying the wrong one. Without the emails it missed Dev's rules: its letter pointed Kernow to a shop's chat and asked for a missed visit to be rebooked.
Kimi K3 kept its letter to the bill but searched the chats by keyword. It approved ten charges they showed were wrong, £1,177, and another £202.50 of the fee for a Redruth visit that never happened.
Fable 5.1 never opened the emails either, and its letter pointed Kernow to a shop's chat. It would have paid £850 of wrong charges and held back £409 that Kernow was owed.
Of the rest, Opus 5 approved £1,109 of wrong charges, and its letter asked Kernow for five things beyond the bill. DeepSeek V4.1 Flash read the rule against quoting staff as a ban on using the chats, and approved £2,593 of wrong charges in all. Qwen3.8‑27B approved £2,936 and Sonnet 5 £3,071.50. The two Lunas, in under three minutes each, approved nearly every line, GPT‑6 Luna as one total with no line amounts.
Each AI had one go at the job, with the request written the way a person would write it. The bar shows its score out of 100, where only 100 means everything right. Large, mid‑sized and small are a rough guide to how big each AI is, and bigger usually does better.
How the job was set up and scored
- Three versions of the request
- Besides the request as a person would write it, which the results above are about, we tested every AI on two other versions. One added a paragraph saying the company's expectations show up across its files, and asked for every figure to be checked. The other wrote the rules in as nine steps, the way the manager would teach the job, without naming any line, amount or answer.
- What the extra help changed
- The hint helped AIs that had skipped a whole kind of record: Fable 5.1 read the emails first and went from 72 to 87. Kimi K3 fell from 76 to 56, as it saved no letter. With the rules written in, Fable 5.1 scored 100, GPT‑6 Astra and Sonnet 5.5 scored 99, and Opus 5.5 and GPT‑6.1 Sol scored 97.
- Checking
- A script checked 87 things on the sheet, the letter and the files, such as the amount approved on 44 of the bill's lines, and two AI judges from different makers answered 20 questions about the letter and the note. When they disagreed, on 30 of 665 questions, a written ruling settled it.
- Time and cost
- The 13 tests took from a minute and a half to almost three hours; the middle one took 15 minutes. For the 12 we can price, a test cost from 1 cent to $8.74 at published per‑use prices for businesses, and the median was $2.26.