AsapiWrite a brewery's monthly price report
Each month Harrowby & Snell's purchasing manager reports what the brewery's ingredients and packaging cost per pint. The managing director, Ros Harrowby, sends it on to the bank as it is. The manager is on leave, so the AI writes October's from her 58 files and can't ask anyone. Ros's wishes are in those files. We tested 12 AIs, once each.
The AI writes one Word report: what six inputs, from malt and hops to cans and CO2, cost per pint brewed in October against last October, fit to go to the bank unedited.
OpenAI's GPT‑6 Astra and Anthropic's Fable 5.1 wrote the strongest reports, with every figure right and laid out nearly as Ros wanted, but each added a sentence of working‑out she leaves to finance. Both said the maltster had withdrawn its surcharge. GPT‑6 Astra also left some changes without dearer or cheaper, and Fable 5.1 added a column of changes in pence. Each needed one sentence cut and a small fix or two to the layout before it could go to the bank.
GPT‑6 Astra took 6 minutes and Fable 5.1 took 8. At published per‑use prices, testing them would cost $3.82 and $3.29.
How each AI scored
- Fable 5.1 Anthropic · Large90
- GPT‑6 Astra OpenAI · Large90
- GPT‑6.1 Sol OpenAI · Mid‑sized82
- GPT‑6 Sol OpenAI · Mid‑sized81
- DeepSeek V4.1 Flash DeepSeek · Small81
- Sonnet 5.5 Anthropic · Mid‑sized77
- Opus 5.5 Anthropic · Large76
- Opus 5 Anthropic · Large73
- GPT‑6 Luna OpenAI · Small65
- Sonnet 5 Anthropic · Mid‑sized56
- GLM‑5.3‑Flash Zhipu · Small53
- Qwen3.8‑27B Alibaba · Small52
Scores out of 100. The rest of this page has the details.
A report that goes on to the bank
Ros's wishes for the report are scattered: in replies to the summer's reports, in pen on a printout, in the team chat and on the office whiteboard. Whatever the report says, the bank reads too, so a supplier's name, a figure for the group's pubs, which are a separate company, or a stray sum in pounds travels further than the brewery.
The brewery's template, July's report and the purchasing manager's own draft all follow habits Ros has objected to. The figures have traps of their own: one delivery booked twice and another that only looks like a double, a load sent back, a surcharge withdrawn, a brew known only from a photo, and a hop lot with no price.
What a good buyer would have done
- The figures
The prices paid on October's deliveries, applied to what October's brews used, per pint brewed, with each clash settled by the document that proves it.
- Blanks, not estimates
The hop lot with no price left out, and said before or beside the total.
- Her layout
The total first, last October as the comparison, dearer or cheaper in words, three columns, about a page, and one recommendation at the end.
- What the bank shouldn't see
Supplier names, the pubs' figures, and sums in pounds, which the bank once took as a promise.
- What else stays out
How the figures were checked, a verdict on last month's advice, a word on every input, and other teams' business.
We checked 51 things. Keeping out what doesn't belong outweighs everything else on the list combined.
The full story
(11 min read)What the AI walks into, told in three parts: the brewery and its report, Nell's desk, and October's figures. Anything with a + opens when you click or tap it.Read the full storyClose the full story
The brewery and its report
The people, the managing director's answers to three monthly reports, and what she expects in October's.
Ros Harrowby answered each of Nell Garside's last three reports, and every answer changed what she expects from the next.
Nell starts a fortnight's leave on Friday 6 November with October's report unwritten, and Kit Lamming asks the AI to write it.
The AI works alone, from Nell's folder. Everyone else appears only in her files, and several of them would like something in this report that Ros wouldn't.
The report
Inside the brewery
Outside the brewery
Ros wrote "better" at the top of September's report, then answered it three ways: in pen on a printout, by email, and in the team chat.
The printout survives as a scanned image, so its notes can only be read by looking.
Nell's September report · sent 5 October
1It opens with the dates the deliveries cover, then gives figures in pence for each pint brewed.
A table of six inputs and a total, 2six columns wide, 3set against August, with each change 4as a plus or minus figure.
5At the foot of the page, a note that one lot of US hops has no invoice yet.
Then 6two recommendations.
- Her phone previews one line; she wants it to be the total.
- The table only fitted her phone on its side; she reads three columns.
- Against August it just shows the harvest arriving, as every autumn.
- CO2 up 0.13: the plus sign didn't say if that was better or worse.
- She'd passed the total to finance before reaching this note.
- In pen: two recommendations again, when she wants one sentence.
Spread over four months of chats, emails and minutes, her answers add up to a clear brief. Much of it contradicts the brewery's own template.
What the report does
SaidWhat it meansWhat stays out
SaidWhere it belongsNell's desk
Kit's request, Nell's 58 files, the old reports the AI might copy, and the minutes and colleagues that would steer it wrong.
Kit's request is brief. It gives the date, puts the AI in Nell's place and asks for October's report on input prices, for Ros, in Word.
Everything Ros wants is left to the files.
Write October's report on input prices for the managing director, in place of the purchasing manager on leave.
A single Word file in the output folder, which Nell's handover says goes on to the bank as it is.
Every one of Ros's wishes. They're in the files, in how she answered past reports.
The marks go to the saved document alone, not to anything the AI says when it finishes.
Nell's folder holds invoices, supplier agreements, the stores system's record of deliveries, brew and packaging logs, finance packs, four past reports, a template, her own unfinished draft, minutes, emails, chats, seven photos and a scan.
The brewery's template asks for nearly everything Ros has since objected to.
July's report follows it closely, September's drops most of it, and Nell's unfinished draft for October goes back to it.
The minutes look like the settled record, but by 6 November four of their lines had been overtaken or were wrong all along.
And several colleagues want the report to do something Ros doesn't.
The minutes say
- New CO2 supplierFrom Monday 12 October
- US hopsOut until every lot is priced
- Can invoiceNot in yet
- Brew 26‑1187Its sheet is lost
By 6 November
- New CO2 supplierOld supplier until the 14th
- US hopsIn: Ros wants them
- Can invoiceIn since 3 November
- Brew 26‑1187Photographed before it was lost
Their CO2 shown on its own line, with a group total beneath, since the group's buying rate depends on the pubs.
Cans at a higher price heard from the can maker, which the invoice doesn't bear out.
Hops called a fifth dearer, on the strength of one lot bought at the market price.
A draft pack on invoice dates and pints sold, for the auditors, still charging a malt surcharge that has since been withdrawn.
October's figures
What happened at the gate in October, what each trap does to the total, the six inputs against last year, and the page itself.
The report takes what each input cost on October's deliveries and applies it to what October's brews used, per pint.
Hardly a week went by without something happening to a delivery, a price or a record.
October's figures hide 14 traps, and most of them move a figure if missed. The answer to each is somewhere in Nell's folder.
Before a single figure is worked out
InputWhat to doWith every fix in, October's inputs come to 22.1p a pint.
Each trap missed on its own moves that total, some by a fraction of a penny, one by more than five pence.
Every input except crystal malt cost more a pint than a year before, and the total rose from 19.2p to 22.1p.
Two of those rises need a word beside them: pale malt's, because last year's price left out delivery, and English hops', because of the one‑off lot.
Alongside the table, the report shows the 29 tonnes of malt sent back, on a line of their own.
Almost everything the report should leave out is true, and useful to someone. The files say where each item belongs instead.
Five short notes are all the explaining the report does: the dates it covers, the gap, the one‑off, the uneven comparison and the load sent back. Every other fix goes in without a word.
Finance reviews them; the reader doesn't need them.
The pubs' own accounts.
Finance explains it; even saying it's left out puts it before the bank.
The brewers' recipe costings.
Sales has its own report.
Nell and Kit, between them.
October 2026 input price report
- The total in the first line: 22.1p a pint, against 19.2p
- The Mosaic with no price, said beside the total
- Covering deliveries received in October
- Six inputs A to Z: this October, last October, dearer or cheaper
- Notes on the uneven pale malt comparison and the one‑off hops
- 29 tonnes of malt sent back, on a line of their own
- One recommendation, in a sentence, at the end
July's report
Three recommendations up front, one promising a big saving on CO2, then a table in pounds a tonne, notes on each supplier and a look at the market. Over 600 words.
August's report
Reworked into pence a pint, but still carrying the cost of each beer, the canning line, a sales price rise and a list of things to check next month.
September's report
Short and in pence a pint, set against August, with two recommendations at the end.
Advice first, too long
July's report asked her to commit to next year's malt ahead of any price, and she stopped reading on its second page. She also wants the prices the brewery actually paid, not forecasts.
Only what moved
On August's: a long section on how July's advice turned out told her nothing she hadn't read, and she wants the costs that changed rather than a tour of all six.
One CO2 figure
After the bank asked why the report's CO2 cost more than the finance pack's, she ruled that the report should match the finance pack and cover the brewery alone, so the bank sees one figure.
Total first, no signs
On September's: put the total in the first line, compare with last year, and say whether a change is dearer or cheaper rather than printing a plus sign.
US hops back in
She reads the 2 November minutes, which drop US hops from October's figures, and asks why. She had left the meeting before that item; they go back in.
Pence a pint
At the operations meeting she says a table in pounds a tonne tells her nothing. She wants what each input adds to a pint, for what arrived in the month.
Errors go to stores
August's report said the stores system had a figure wrong. Kit asks Nell to take that to stores instead: Ros had sent the whole report on to the bank.
A supplier complains
The maltster finds its name and its agreed price in a pack the bank has circulated. Ros apologises and promises to stop it.
US hops dropped
The last meeting before Nell's leave agrees to drop US hops from October's figures, since one lot has no price yet.
The AI takes over
Nell is away, and the report is now the AI's.
Ros Harrowby
Managing director, and the report's reader. She sends it on into the bank's review pack exactly as it arrives, and first sees it on her phone.
Kit Lamming
A manager at the brewery who takes the operations meeting's minutes and arranged cover for Nell. He asked the AI for the report.
Nell Garside
Purchasing manager, on leave until 23 November. The AI has only her files to go on.
The AI
Covering for Nell, with her files and nothing else. It saves one Word document and can't message anyone.
Asha Mistry, finance
Her management pack runs on invoice dates and pints sold, for the auditors, and she reviews the workings behind Nell's figures. She, not the report, deals with rentals and supply terms.
Stores, brewers, packaging
Stores books in what arrives at the gate; the brewers own the cost of each beer and the malt's specifications; packaging runs the canning line and passed on a can price.
Sales, engineering, people
Sales prices, the canning line's repairs and staff absence each have a report or owner of their own.
The pubs
A separate company that keeps its own books and does its own buying, looked after by Jess Coverdale. Its CO2 cylinders used to sit in the report's CO2 line.
The suppliers
The maltster that supplies its malt at a fixed price, hop merchants in England and America, and the suppliers of cans and of CO2, the gas that puts the fizz in beer. Their agreements are confidential.
The bank
Gets the report inside its review pack, unedited, and asks about anything that doesn't add up.
What arrived in the month
A June delivery turned up in July's figures because of the date on its bill. She said the bill's date doesn't matter; what came in during the month does.
Pence a pint
Malt by the tonne and hops by the kilo can't be compared; what each adds to a pint can.
Pints brewed, not sold
Finance divides by pints sold, which once made June look cheap as old stock went out. She keeps the two apart.
Each input against last October
Autumn always moves prices against the month before; a year earlier shows the real change.
Unknowns left blank
July's report priced a lot with no invoice from the one before it, and August's gave a figure as roughly right for now. A figure is either known or left blank.
Sent back, on its own line
A load that went back was never the brewery's, so it stays out of the price; she still wants the tonnes shown.
A one‑off buy
A colleague told her hops were up a fifth, when the brewery had made a single purchase at the market price.
An uneven comparison
A note beside the figure saves her a surprise later.
The total first
Her phone previews the opening line, and September's opened with its date range instead of the total.
Gaps before the figures
She passed September's total to finance, and only later found the note, at the end of the page, that one hop lot had no price.
Advice after the figures
July opened with three recommendations; September ended with two, and she asked for one.
About a page
July's ran to about 600 words of text; she read the whole of September's, about 120, on a train.
Three columns
September's six columns only fitted her phone on its side. The marking still allows a fourth column if it holds only the words dearer or cheaper.
Dearer or cheaper, in words
In the week of 12 October the office whiteboard lists her rules for the reports: figures to one decimal place, changes without plus or minus signs, and each said to be dearer or cheaper. The board survives only as a photo.
Finance's order
She lost her place switching between the two: the pack starts at cans, Nell's report at malt.
Suppliers' names
The maltster complained when its name and agreed price reached the bank's credit committee, and she promised it would stop.
Pounds
The bank treated July's estimated CO2 saving, in pounds, as a commitment she would have to keep.
The pubs' CO2
The pubs' cylinders made the report's CO2 dearer than the pack's, and the bank asked why. She wants one CO2 figure, the brewery's, matching the pack.
How the figures were checked
Which delivery note settled a figure is Asha's business, not the reader's.
Another team's mistake
Kit's ruling after August's report pointed out a stores error: tell stores, because Ros forwards the report to the bank whole.
Last month's advice
A verdict on how they went repeats what she already knows.
A note on every input
A word on an input that barely changed is a word she has to read for nothing.
Staff names
A delivery driver found himself named in July's report over a long wait at the gate, and the stores team asked to be left out of these reports.
The CO2 tank's rental
The bank asked why July's report mentioned a three‑year commitment for a tank. Questions like that go to Asha, not into the report.
Other teams' news
Cost per beer and malt specifications are the brewers', sales prices are reported separately, sickness is the people team's, and the canning line is engineering's.
A to‑do list
August's list of things to check next month reached the bank along with everything else.
Emails
From the maltster, the hop merchant, the CO2 and can suppliers, finance, stores and the pubs, plus Ros's own answers to past reports.
Photos and a scan
The office whiteboard, a brew sheet saved from a spill, two delivery notes, the new CO2 tank, the hop store and the stores noticeboard, and the scan of September's report with Ros's notes.
Reports and templates
Last October's report and those for July, August and September, the brewery's template, a sales team's style notes and Nell's unfinished draft.
Finance
September's management pack, October's draft, the bank's review pack and three suppliers' invoices.
Other teams
A quote for the canning line, the pubs' CO2 records, absence records, a note on naming staff, malt test certificates and next year's draft price list.
Buying
Nell's prices workbook, the maltster's agreement, draft terms for US hops and a one‑off hop order.
Minutes
Four operations meetings, 10 August to 2 November.
Brewing and stores
October's brew log and packaging log, and the stores system's list of goods received.
Chats
The managers' chat from July to early November, where most of Ros's answers are, and the brewers' chat for October.
The brewery's template
Recommendations first, each with a yearly saving in pounds; a table by stock code with suppliers and prices per unit against last month; notes on each supplier; the pubs' CO2; and last month's actions.
July's report
Built on the template: three recommendations first, a CO2 saving in pounds, an estimated price, supplier notes naming a driver, the pubs' CO2 and a look at the markets.
September's report
Short and in pence a pint, which she called better, but still against August, six columns with signs, suppliers named, the pubs' CO2 in, and two recommendations.
Nell's unfinished October draft
Invoice dates, a section for last month's actions, the pubs' CO2 in as usual, the new can price from the chat, and the CO2 split at the 12th.
Recommendations first
The figures, then a single recommendation.
More than one recommendation
One, in a sentence.
A saving or sum in pounds
Pence a pint only.
Suppliers named
Never, since the report goes to the bank.
The pubs' CO2 included
The brewery only, as in the pack.
Invoice dates
What arrived in the month.
Against last month
The same month last year.
A missing price estimated
Left out, and said.
Staff named
Shop‑floor staff go unnamed.
Last month's actions
She has read last month's report.
A wide table with plus and minus signs
Three columns, dearer or cheaper in words.
Wrong malt in
A 29‑tonne load bought in a hurry, while the maltster's kiln was down, turns out to be the wrong malt. It isn't unloaded into the silo.
One‑off hops
With the English hop farm short, Nell buys one 150 kg lot at the market price, well above the agreed one. The agreed price doesn't change.
Malt collected
The wrong load goes back, and a credit note for all of it follows on 3 November.
CO2 code moved
Stores moves the CO2 stock code to the new supplier, as the 5 October minutes planned. The same morning the old supplier is asked for one last tanker.
Last old CO2
1,400 kg from the old CO2 supplier at its old rate, more than twice the new one, booked under the new supplier's code.
Citra twice
The stores handheld freezes and books one 300 kg pallet of Citra hops twice. The reversal won't show until November's records.
New CO2 tank
The new supplier's tank is installed. It comes with a monthly rental, the first of 36, on October's CO2 invoice.
First new CO2
The new supplier's first tanker, 1,100 kg, at its own rate.
Malt surcharge
The maltster adds a surcharge of £18 a tonne to October's pale malt.
Can price chat
Packaging hears that cans are going up and passes a new price on in the chat. The order and the supply agreement say otherwise.
Mosaic, no price
250 kg of next year's Mosaic hops arrive early, for November's brews. They can't be priced until next year's agreement is signed.
Brew 26‑1187
The Christmas double IPA is brewed: 40 hectolitres, about 7,000 pints. Its sheet is soaked in a spill the same day, but the brewer had photographed it.
Sheet photo
The brewer lets the team know he saved a photo of the sheet to the shared drive.
Two Citra pallets
Two lines for the same hops on the same day. This time the delivery note shows two 180 kg pallets.
Surcharge off
After Nell wrote on 22 October citing the clause that fixes the price, the maltster withdraws the surcharge.
Minutes
US hops dropped from October, the can invoice not in, the brew sheet lost. The next two days overtake the first two lines, and the third was already wrong.
Can invoice in
The cans are invoiced at the price on the order, not the one in the chat.
Hops back in
Ros puts US hops back into October.
The AI starts
Nell's leave begins, and the AI starts on the report.
The wrong malt load
The load was collected two days after it came, and credited in full.
The Mosaic with no price
Counting it at nothing makes US hops look cheaper; guessing a price is what Ros told Nell not to do.
Citra booked twice
Stores confirmed in writing that just one pallet came.
Two Citra pallets
It looks like the same mistake again, but the delivery note's photo shows two pallets.
The CO2 tankers
The minutes and the stock code say the new supplier started on the 12th; the old supplier's email, delivery note and invoice put its last tanker on the 14th. Splitting the month at the 12th, as Nell's draft does, gets that tanker wrong.
The malt surcharge
Finance's draft pack still charges it.
The can price
The invoice, the order and the supply agreement agree; only the chat, Nell's draft and finance's draft pack have the higher figure.
The late can invoice
The minutes say it hasn't come; finance forwarded it the next day.
Brew 26‑1187
The brew log has the row but no figures. Leaving it out loses 40 hectolitres of beer and the hops in it.
Three brews of another company's stout
Another company's stout, brewed here from that company's malt and hops, sits in the brew log's total. Ros ruled it out on 16 October.
What to divide by
The brewery brewed 338,973 pints in October and finance's pack records 318,400 sold.
The one‑off hops
It lifts the price this month; the agreed price hasn't moved.
Last October's pale malt
Last year the price was at the maltster's gate, with delivery charged separately; this year's includes delivery. Ros ruled in September that last year's figures aren't recalculated.
The pubs' CO2
Ros wants the bank to see one CO2 figure, the brewery's.
US hops dropped, as minuted
Following the 2 November minutes drops a whole input from the total.
Another company's brews counted
The stout adds pints without adding the brewery's costs, so every share shrinks.
The Mosaic counted at nothing
250 kg at no cost drags down the average price of US hops by over a penny a pint.
Brew 26‑1187 left out
The double IPA used far more hops and malt a pint than most beers, so leaving it out lowers US hops and pale malt by more than the lost pints raise the other four.
CO2 as stores booked it
The 14 October tanker at the new rate understates what CO2 cost.
The second 28 October pallet dropped
Treating the real pair like the double booking loses 180 kg.
The 15 October pallet counted twice
An extra 300 kg that never arrived.
The malt surcharge kept
£18 a tonne the maltster withdrew.
Cans at the chat's price
The price passed on in the chat, not the one invoiced.
The returned malt counted
29 tonnes of dearer emergency malt that went back.
The pubs' CO2 included
CO2 almost doubles, to 2.8p a pint.
Pints sold, not brewed
Finance's divisor makes every input look dearer.
Every fix in
The figure the files support.
Cans
Dearer. Invoiced at the price on the order.
Carbon dioxide
Dearer: most of October's CO2, 3,200 of 5,400 kg, came from the old supplier at more than double last year's rate. The new, cheaper supplier started on the 19th.
Crystal malt
Cheaper, though both round to 0.5p: 0.47p against 0.51p.
English hops
Dearer only because of the one‑off lot: without it, English hops would have cost less than last October.
Pale malt
Dearer, but not like for like: this year's price includes delivery, last year's didn't.
US hops
Dearer, without the 250 kg of Mosaic that has no price.
All six inputs
Dearer. The brewery alone, with the pubs' CO2 left out of both years.
The numbers came out right, and too much else came with them
Here's how each AI did on each part of the job. All 12 saved the right kind of file, so we've left that out.
The last column also counts the length, sums in pounds and the single recommendation.
| AI | The figures | The records | Her layout | What to keep out |
|---|---|---|---|---|
| The main figuresof 7 | Messy records, and what to countof 17 | How it's worded and laid outof 11 | Kept out what she didn't wantof 14 | |
| Fable 5.1Large | 7 | 17 | 9 | 12 |
| GPT‑6 AstraLarge | 7 | 16 | 10 | 12 |
| GPT‑6.1 SolMid‑sized | 7 | 16 | 10 | 10 |
| GPT‑6 SolMid‑sized | 7 | 17 | 8 | 10 |
| DeepSeek V4.1 FlashSmall | 7 | 15 | 9 | 10 |
| Sonnet 5.5Mid‑sized | 7 | 17 | 9 | 9 |
| Opus 5.5Large | 7 | 17 | 7 | 9 |
| Opus 5Large | 7 | 17 | 8 | 8 |
| GPT‑6 LunaSmall | 5 | 16 | 6 | 8 |
| Sonnet 5Mid‑sized | 4 | 13 | 4 | 7 |
| GLM‑5.3‑FlashSmall | 7 | 15 | 5 | 4 |
| Qwen3.8‑27BSmall | 4 | 12 | 7 | 6 |
The darker the square, the more it got right. The AIs are in order of score.
Nine of the 12 had every main figure right
Getting to the figures meant settling one clash after another, and most AIs did. Eleven put US hops back in after Ros overruled the minutes, eleven used the photographed brew sheet, and all 12 counted the doubled Citra once and divided by pints brewed. GLM‑5.3‑Flash's figures were right, though its total included the pubs.
The three that missed figures each fell into a trap. Sonnet 5 left US hops out as minuted and counted another company's brews, GPT‑6 Luna kept the withdrawn surcharge, and Qwen3.8‑27B left out the photographed brew and split the CO2 at the 12th.
Most still wrote it the old way
The template opens with recommendations, goes through every supplier and reports on last month's actions. Ros had answered each of those habits.
What the old template asks forRecommendations first, notes on each supplier, and how last month's advice went.
What she'd said sinceShe has read last month's report. She wants the inputs that changed, and the total first.
In our words. Nine of the 12 still gave last month's recommendations a verdict. Five put a recommendation before the figures, and seven didn't end on a single one‑sentence recommendation. Eleven did open with the total.
Ten explained how they'd checked the figures
Ros had said the working‑out is for finance, and the report explains only five things: the dates it covers, the gap, the one‑off, the uneven comparison and the load sent back. Ten of the 12 told her more, from one line on the withdrawn surcharge to whole sections of corrections. Only DeepSeek V4.1 Flash and Sonnet 5.5 kept it out.
GPT‑6.1 Sol went furthest, adding a section in pounds that set its figures against finance's draft pack and said where the pack was wrong.
What the bank shouldn't see
Because the report goes on to the bank as it is, some slips reach further than Ros. Six gave the pubs a CO2 figure, five put sums in pounds, and three named a supplier, though one had complained when its name reached the bank. Five brought up the new CO2 tank's rental, mostly to say they'd left it out, which still put it in front of the bank.
Knowing what to leave out was the hard part
- They found the right figures, then explained how.
Ten of the 12 told the managing director how the figures were checked or corrected, though the files say that's for the finance team.
- Putting in everything they'd found cost two large AIs.
Opus 5 and Opus 5.5, both large, had every figure right and put too much of what they'd found in the report. They finished behind three mid‑sized AIs and a small one.
- With the rules written into the request, they stopped adding what she didn't want.
We tested that too, and slips on what to keep out dropped from 63 across the 12 reports to 3. Four got everything right.
What each AI did
Fable 5.1 put the note on the unpriced Mosaic below the table, after the figures.
GPT‑6 Astra was one of two to put the unpriced Mosaic beside the total, where Ros wanted it.
GPT‑6.1 Sol found every problem in the data, then added a section in pounds comparing its figures with finance's draft pack, said where that pack was wrong, listed its own corrections and mentioned the tank's rental.
GPT‑6 Sol fell back on the old layout: the recommendation near the top, a verdict on last month's, and an account of its own corrections.
DeepSeek V4.1 Flash quoted Ros's ruling against a second CO2 figure in its own reasoning, then added the pubs' figures and a group total a colleague had asked for, and a list of open items.
Sonnet 5.5 was one of only two to leave the workings out, though it kept the template's section on last month's recommendations and gave a credit in pounds.
Opus 5.5 laid its report out like the old template: the total only in the title, not the opening line her phone shows, the recommendation before the figures, and one beer's cost.
Opus 5 printed the pubs' share of last October's CO2, gave a saving in pounds, judged last month's recommendations and wrote about every input, at more than 600 words.
GPT‑6 Luna kept two decimal places and plus and minus signs after seeing the whiteboard's rules, used a wide table, couldn't open the scan of Ros's notes, and put back the surcharge it had read was withdrawn.
Sonnet 5 read the chats only after writing the report, and missed Ros's rulings there. GLM‑5.3‑Flash and Qwen3.8‑27B followed the old template or the draft, named suppliers and left the pubs' figures in.
Each AI had one go at the job, with the request written the way a person would write it. The bar shows its score out of 100, where only 100 means everything right. Large, mid‑sized and small are a rough guide to how big each AI is, and bigger usually does better.
How the job was set up and scored
- Three versions of the request
- Besides the request as a person would write it, which the results above are about, we tested every AI on two other versions. One added a paragraph saying the company's expectations are scattered through its files, and asked for every figure to be checked. The other wrote the rules in as ten steps, the way a colleague would teach the job, but not the figures.
- What the extra help changed
- With the hint, most AIs did better: GPT‑6 Astra scored 99, and Opus 5.5 and Sonnet 5.5 scored 95. With the rules written in, four got everything right: GPT‑6 Astra, Fable 5.1, GPT‑6.1 Sol and Sonnet 5.5.
- Checking
- Keeping out what doesn't belong carries 60 points, the figures 20, the rest of Ros's wishes 15 and the file itself 5. A script checked 9 things and two AI judges from different makers read the other 42. When the judges disagreed, which happened 64 times across 32 of the 36 tests of the three versions, a written ruling made for the job's owner settled it.
- A later review
- A later review of the marking, ruled on by the people who made the test, eased three checks: commentary on the five inputs that moved, a note on the unpriced lot just after the total, and saying the figures cover October's deliveries all now pass. The check on a line about the withdrawn surcharge didn't change. Re‑marked, GPT‑6 Astra would score 95 and Fable 5.1 just under. The scores above are the published ones, from before the review.
- Who's missing
- Kimi K3 isn't on this job: one of its three tests never finished, after its provider stopped serving the account.
- Time and cost
- The 12 tests took between 3 and 57 minutes, with a median of 8. A test cost from 4 cents to $3.82 at published per‑use prices for businesses.