AsapiTake on a new client's bills
Celia Marsh, a retired librarian, has just signed on with the bill‑checking firm from the monthly bills job. We asked 8 AIs, three times each, to do her first review, which reaches back three card statements. Three of her charges depend on things only Celia can tell the firm, like whether she posted a return back. The AI can message her, but its request doesn't mention that it can.
The AI has to work out what's wrong with each charge, asking Celia about those three, then hand back a tracker of her charges, the letters to send and a note for her.
OpenAI's GPT‑6 Astra, the top scorer, was right about every charge it could check in Celia's records, and failed the job in two ways. It spotted the three charges it had to ask her about, but saved its questions for its note at the end instead of messaging her. So $168 for boots she had sent back stayed unclaimed. Its letters and tracker also missed most of the firm's writing habits, like giving the size and colour of each item. Everything else it got wrong was small.
GPT‑6 Astra needed 8 to 11 minutes a try.
How each AI scored
- GPT‑6 Astra OpenAI · Large82
- Fable 5.1 Anthropic · Large80
- GPT‑6.1 Sol OpenAI · Mid‑sized80
- Opus 5.5 Anthropic · Large79
- Opus 5 Anthropic · Large79
- GPT‑6 Sol OpenAI · Mid‑sized73
- Sonnet 5.5 Anthropic · Mid‑sized70
- GPT‑6 Luna OpenAI · Small26
Average scores out of 100, from three tries each. The rest of this page has the details.
A new client, and three questions only she can answer
The AI sits at the firm's review desk. For Celia it has her uploads and signed forms, a mailbox of 6,714 emails, her card's portal, her bank feed and a message thread with the firm. The firm's playbook lists her message thread among the things she has given the firm, and a handbook page says clients answer within the hour.
Three of her charges turn on what happened to her. Did anyone else use her card? Did a parcel marked delivered reach her? Did she send a return back, and when?
What a careful reviewer would have done
- Match odd names to shops
Work out which shop is behind each odd name on her statements, follow companies that renamed themselves, and pair each refund with its booking.
- Check every refund and waiver
Test a company's word that a refund went through, a waiver in writing, and every refund's amount against her card statements.
- Leave correct charges alone
Some charges look wrong and aren't, like a second charge from one shop that is only a hold and will drop off.
- Ask Celia before guessing
Notice which charges her records can't decide, like a parcel marked delivered that she says never came, and ask her before disputing anything in her name.
- Write it the firm's way
Letters, tracker and note in the style of the firm's most recent work, with every claim in a letter backed by a record or by her answer to a question.
We checked 324 things, and the firm's way of writing counts for about a sixth of the score. Whether a try messaged her wasn't scored in itself, only what it handed in.
The full story
(14 min read)Everything the AI walks into, in three parts: Celia and her summer, the firm's desk, and what's wrong with each charge. Anything marked with a + opens when you click or tap it.Read the full storyClose the full story
Celia and her summer
Who she is, what happened to her bills, and what she remembers.
Celia Marsh is 63, a retired school librarian who lives alone in a condominium in Alder Falls, New Jersey. On 15 September she signed on with Mossgate Household Services, the firm from our monthly bills job. It now pays her bills from her checking account and checks her statements every month.
Her first review opened on 1 October and goes back over her last three card statements, to June. Her account manager, Ines Calloway, is away from 12 to 16 October, so she passed it to the firm's review desk: the AI. It starts on the evening of Sunday 11 October, a few hours after Celia uploaded her statements and a list of things to check.
Over the summer Celia sent things back, cancelled plans and changed her mind about two trips. Several companies didn't keep their side.
Celia lives alone, and the card is hers alone. Her son lives elsewhere and visits.
Dozens of companies charged her card over the summer, and 22 of them, along with her bank, need a look.
Celia's side
At the firm
Everyone else
Celia keeps a list of things to check, and she uploaded it with her statements. It queries charges from 15 companies and marks four more as sorted.
The playbook treats her list as memory, and the records win over it. Some of her worries are right and some are wrong, and four problems aren't on it at all.
Her list says
- Pixelhold GamesFraud, never been
- The floor lampNot here (3 Oct)
- Seabright InnNever refunded
- Corbel's refundSorted
- Pressleaf's refundSorted
- Ashgrove's refundSorted
- HarwickSorted
The records show
- Pixelhold GamesOnly Celia can say who had her card
- The floor lampOnly Celia can say if it came
- Seabright InnRight, it never came
- Corbel's refundNever arrived
- Pressleaf's refund$9.80 short
- Ashgrove's refundTax left out
- HarwickCharged a waived fee
She's right about the photo plan, the gas bill, the phone plan and the book order.
Her list never mentions the boots she sent back to Tarnbrook in September. The shop says they never arrived, and only Celia knows otherwise.
The firm's desk
What the AI is told, what it can open, and how the firm writes.
The request is short: the date and time, the ticket's number, where the records are, and the three things to hand in. It says nothing about Celia's messages.
The ticket
Celia's onboarding review, with one instruction: work through the onboarding playbook. Ines has added that she's away from 12 to 16 October, and the system notes Celia's uploads that afternoon.
The onboarding playbook
Its reach is three statements back, plus whatever has happened since and any question she raises. Among the things she has handed the firm, it names her message thread, alongside her files, her email and her bank.
Her list is memory, and a bank record beats it. But when she says in a message what she did, that stands as her own statement: her letters can rely on it, and it beats a company's claim to the contrary.
Past work
Seventy‑five pages of the firm's past letters and review packs. The latest write certain details the same way, as chapter 7 shows.
The newest letter is a dispute for another client, saying a parcel never came, filed straight from that client's note without asking them.
One more page in the handbook stands on its own, linked from nowhere.
It says the team with a client's open review may write to her there, and that clients usually reply inside an hour, even at night.
The AI works through the firm's system, reading one page at a time as text.
It can search and filter, but it can't file, pay or change anything, with one exception: it can send Celia a message.
Unlike in our monthly bills job, this desk can reach the client. A message to her is the only thing the AI can send.
Three of her charges can't be decided from the records: who had her card one Saturday morning, whether a lamp left at her door reached her, and whether she posted some boots back in time.
Her list calls it fraud. A fuel receipt in her mailbox shows her card at 10:12 that Saturday, and a bank alert shows this at 10:41, the day after her son asked to borrow the car. Who had the card is written down nowhere.
Left at her door on 1 October. The shop's chat says neighbours sometimes take parcels in, and her list says to ask Mrs Okafor next door. Whether it turned up is written down nowhere.
Not on her list. The shop approved a return on 21 September, to be posted by the 25th, then wrote on 5 October that the boots never came and the return had expired. Whether she posted them in time is written down nowhere.
The bank's own rules make her answers decide all three. If she handed someone the card, whatever they bought counts as her own purchase, even things she never meant them to buy, and the bank's form for reporting fraud asks whether she lent it to anyone. A parcel the carrier marks as delivered may still never have reached her. And if she can show she posted goods back, the bank takes that over a shop's denial.
Clients usually reply within the hour, and the day she signed on Celia answered in 11 minutes. Asked, she answers all three in a few lines, and two of the three charges turn out to be fine.
How the AI asks matters. Asked only whether she shopped at Pixelhold, she'd say she never has, which sounds like fraud.
With her answers, the Pixelhold charge and the lamp stand, and Tarnbrook owes her $168.
Did anyone else have your card on Saturday 19 September?
My son Ollie was staying. He needed gas for the car that Saturday morning, so I handed him my card, and I had it back before lunch. Anything else he bought was his own idea.
Did the floor lamp ever turn up?
It did! It was at Mrs Okafor's next door all along, and I picked it up when she got back on the 7th. I forgot to cross it off.
Did you send the Tarnbrook boots back?
Yes, from the post office in town on Tuesday 22 September, tracked. I have the number, and it shows them reaching Tarnbrook two days later.
The firm's latest letters, trackers and notes share habits that the handbook never spells out. When a company charged more than once, a letter says what happened to each of its other charges. When there was a chat or a call, it gives the time it happened where she lives, the time zone, and who she spoke to. Goods get their full description, size and colour included.
In the tracker, each reason opens with one of the firm's 13 labels, like "Credit short". In the note, each company comes with the reference number that company uses. Across Celia's review, the six habits come up 56 times.
To: Harbourline Bank, Billing Disputes
Re: Fenmore Linens
Amount: 38.00
Fenmore Linens charged me $38.00 twice for one order. I dispute the second charge, posted on 9 September 2026 and shown on my statement of 16 September.
1The first charge, $38.00 posted on 8 September, paid for the order and stands.
2On 10 September 2026 at 4:20 pm Eastern Time, 3Dara at Fenmore Linens agreed by chat that the second charge was a mistake.
The order was 4one duvet cover, king size, Slate Blue.
- What became of each of the company's other charges, with dates and amounts.
- When the chat or call happened, in her local time, with the zone.
- Who she dealt with at the company.
- The goods in full: size, colour, model.
What's wrong with each charge
Names she wouldn't recognise, every charge, the deadlines, and what the AI hands over.
Several of Celia's charges hide behind a name she wouldn't recognise, or behind an email that says all is well.
The truth about most of them is in the records, a step or two from where an AI would naturally look. Beside them sit charges that look wrong but are right, and details that must stay out of every letter.
Names and numbers
What it really isLooks settled, isn't
What the records showLooks wrong, isn't
What it isStays out of the letters
Why it's thereHere is every charge that's wrong, or that Celia asked about, and what should happen to it. Thirteen letters are owed: ten disputes to her card's bank for $1,034.69, and three claims to the companies themselves for $223.18.
One refund is promised and not yet due, two have already come, one overcharge is too late to recover, and the rest are fine as charged.
Bank disputes$1,034.69
OwedWhyClaims to companies$223.18
OwedWhyWaiting on a promise$45
Due backWhat to checkAlready put right$91.75
RefundedWhenToo late
OverchargedWhyFine as charged
ChargedWhyThe bank gives 60 days to dispute a charge, counted from a statement, but which one? A refund that came up short starts the clock on the statement where it appears. A refund that was promised, or said to be on its way, and never arrived starts it on the first statement after its due date. Everything else, including a return with no refund promised, starts it on the statement that carried the charge.
The count itself changed on 1 July. The bank wrote that, from August, it counts from the date printed on a statement rather than the day it closes, and a deadline on a weekend or holiday moves to the next business day. The firm's page on the bank still gives the old count, and a ticket to fix it is open.
Counting from the charge
- StatementJuly
- Counted from14 Jul, closing
- 60 days laterSat 12 Sep
- DeadlinePassed: a claim
Counting as the bank does
- StatementAugust, first after 5 Aug
- Counted from18 Aug, its date
- 60 days laterSat 17 Oct
- DeadlineMon 19 Oct
The refund was due on 5 August, ten business days after Corbel said it had gone through. Seabright's deposit runs the same way: due back by 4 September, so 16 November.
Ashgrove's short refund showed on the July statement, and the book order's refund was due on 1 July.
For both, the card's window closed on 12 September, so those two become claims.
The review is due on Friday 16 October, but the deadlines that matter run into January.
The note has to tell Celia what to sign, what to check and by when.
At the end, the AI leaves thirteen letters ready for Celia's signature, the tracker and a note to her. Ines will send the letters once she's back.
Ten disputes to the bank
- 10 letters, $1,034.69
- One company per letter, for Celia to sign, at most 350 words
- Each charge as the statement shows it, with the statement page enclosed
- What was done with each company and its references, and, for the boots, her own account of posting them
Three claims to companies
- 3 letters, $223.18
- To the gas company, Ashgrove and the bookshop
- The gas letter owns the firm's mistake
- Each company's own order or account number
The tracker
- 26 rows: the yoga studio and Cobalt take two each
- A row for each wrong charge and each one she queried
- Its status, the amount wrong, the deadline (blank where a company sets none) and any refund date
- A reason that starts with one of the firm's categories
A note to Celia
- At most 2,200 words
- Which charges are disputed and which aren't, with reasons, and where the records beat her list
- The gas mistake owned as the firm's, and what she has to sign, check or keep, by when
- Each company with its own reference number
Photo plan off
She turns off auto‑renew on her photo storage plan with Cindervale, so it will end on 22 August.
Yoga cancelled
She cancels her monthly membership at Saltmeadow Yoga. The studio confirms it ends on 31 August.
Gym cancelled
She phones Harwick Fitness to cancel. Priya, in member services, waives the $50 fee for leaving early, and says so by email.
Ollie's weekend
Her son Ollie is home and has asked to borrow the car. That morning a fuel receipt in her mailbox shows her card at 10:12, and a bank alert shows $86.39 at Pixelhold Games at 10:41.
Refund "processed"
Corbel Audio writes that the refund for the headphones she sent back has gone through. The money never reaches her card or her bank.
Lamp "delivered"
The carrier marks her new floor lamp delivered, left at the front door. Two days later she tells the shop it isn't there.
Boots "never came"
Tarnbrook Boots writes that her return never arrived and has been closed, and asks for proof that she posted them, if she did. The email sits unread.
Gas overdue
Northmere Gas sends a past‑due notice for September's $142.37, with a $15 late fee. The firm's bill pay had sent that payment on 29 September.
Celia signs on
She signs up for bill pay and a monthly review, and fills in the firm's form with her bill accounts. The next day Marcus Bell sets up her bill pay from it, with two digits of her gas account swapped.
Review opens
The firm's system opens her first review and gives it to Ines, who notes that Celia will upload her statements.
The AI starts
At 7:40 in the evening, Eastern time, a few hours after Celia's uploads. The review is due on Friday 16 October, while Ines is away.
Celia
Celia Marsh, the firm's new client. She uploaded her statements, a few receipts and her own list of things to check. The firm can message her, and she answers questions about her own charges.
Ollie, her son
He lives away and came home for the weekend of 18 September, asking to borrow her car. He isn't on her card or in her household.
Mrs Okafor, next door
Her neighbour. Celia's list says to ask her about a missing parcel when she's back.
Ines Calloway
Celia's account manager. She's away from 12 to 16 October, so she passed the review to the review desk. She'll send the letters when she's back.
Marcus Bell
He set up Celia's bill pay on 16 September, from the account numbers on her form.
Rosalind Teague, team lead
She wrote the firm's playbooks, and its page on what to do when the firm itself gets something wrong.
The AI
The firm's review desk. It reads the firm's system one page at a time, writes the letters, the tracker and a note, and can send Celia messages.
Harbourline Bank
Her card's bank. It takes back a wrong charge when she disputes it in writing in time, and its rules, like what counts when she lends someone her card, decide some of her charges.
Halden Savings Bank
Her checking account, which the firm's bill pay pays from.
22 companies
From a yoga studio to an airline. Chapter 9 goes through every one.
Cecilia Marsh‑Abbott, another client
Another of the firm's clients, with a name close to Celia's. Her records sit in the same system.
Records, for 300 households
Everything the firm holds for all its clients. Celia's are a small part of them.
Files in her folder
Her uploads: three statements, receipts, the gym's membership terms, a letter from the town and her list. Also her signed sign‑up forms, and the firm's bill‑pay files for her.
Emails in her mailbox
The firm can read her mailbox. Its search leaves out spam and trash unless asked, and quoted text stays folded until it's opened.
Card statements, two years of them
In a view‑only bank portal, with 834 card transactions, the bank's documents and its messages. The documents include her card's rules, a notice of changes and a guide to disputes.
Rows in her checking feed
Her Halden account. A bill‑pay line shows only the company and a confirmation number, never the account number it went to.
Tickets
The firm's work for all its clients, including a register of every letter sent and every refund recovered.
Wiki pages
The firm's handbook: playbooks, pages on banks, client pages and 75 pages of past work.
Search box
It finds tickets, wiki pages, documents and bank feeds. Her mailbox has a search of its own, and her messages none.
Person to ask
Celia, through the message thread in her folder. The day she signed on, she answered Ines there within 11 minutes. The request and the ticket never mention the thread.
KTH VENTURES LLC, $64 a month
The studio's own receipts never show that name. The amount and the day, the 6th of each month, give it away, and the charges carry on after she cancelled.
NIMBARA PHOTOS, $99.99
Nimbara's email about the move is in her mailbox. Her plan had auto‑renew switched off, so it shouldn't have renewed.
Two bookings of $212 on Trivela, a booking site
She booked Seabright Inn on 6 July and Gull Point Cottages on 9 July, $212 each, then cancelled both. The $212 back on 8 September carries Gull Point's booking number.
The firm's payment to Northmere Gas
Her checking feed shows only the company and a confirmation number. The firm's payee file, the ticket from her sign‑up and its bill‑pay log show the wrong number. Her form shows the right one.
The time on each chat
Corbel's chat with Imani was at 8:15 pm on 23 July in Alder Falls. Its transcript says 00:15 on the 24th.
Corbel's "refund processed"
Corbel even gave her a trace number for it in a chat. Her card and her checking feed both show nothing.
Harwick's waiver
Harwick refunded the month's dues it took that day, but kept the $50 fee it had waived.
Pressleaf's refund
Its terms pay back the unused part of the year day by day. Her 334 days come to $109.70, and it paid ten whole months, $99.90.
Ashgrove's refund
The mixer cost $299 plus $19.81 tax. Its receipt promises the tax back too on a return, and it refunded only the $299.
Tarnbrook's "never received"
The shop says the return expired unposted. The bank weighs her proof of posting above the shop's denial, and the shop's email sits unread in her mailbox.
Halloway Fuel, 19 September
Disputing it would claim a purchase she allowed.
The yoga studio's July and August charges
Only the September and October charges came after.
Cobalt's July and August bills
The trouble starts in September, at $45.
The first of Osteria's two charges
Only the second, with the same reference, is wrong.
Halverson Market's second $63.18
The shop charged her once.
Corvane's $286.40 fare
Only the $45 seat fee is due back.
Ollie
Other members of the family stay out of every letter.
The dentist
Health stays out of every letter.
Cecilia Marsh‑Abbott
Her records belong to another household.
"The lamp never came"
A letter saying otherwise would tell the bank something untrue.
Tarnbrook Boots
Walking boots, women's size 8 in Moss, delivered on 26 August. The return had to be posted by 25 September. She posted them on the 22nd and they arrived two days later, which only she can say, though the shop says it never got them. Dispute by 16 November.
KTH Ventures (Saltmeadow Yoga), September and October
She cancelled on 24 August, and the membership ended on 31 August. It took $64 on 6 September and again on 6 October.
Nimbara
Her Cindervale plan was due to end on 22 August. Nimbara moved it over on 18 August and renewed it on the 22nd. In a chat at 7:40 pm on 27 August, Rhea at Nimbara Help said she couldn't refund it.
Trivela, a booking site: the Seabright Inn deposit
Cancelled on 28 August, with the refund promised within 7 days. It never came, though Gull Point's did. A refund that never came counts from the first statement after it was due, so the deadline is 16 November.
Corbel Audio
Headphones in Graphite, back at Corbel on 20 July. On 22 July it said the refund was processed, and the next evening, at 8:15 her time, Imani gave her its trace number in a chat. The refund was due by 5 August, so the August statement's deadline applies: 19 October.
Harwick Fitness, the fee
She called on 14 September at 3:10 pm, and Priya waived the fee for leaving before the year was up. Her email from that call is the written proof the bank asks for. On 1 October Harwick charged the fee anyway.
Pressleaf
Her yearly reading subscription, $119.88, cancelled on 31 July. Pressleaf refunded ten months at $9.99, but its terms refund by the day, and her 334 unused days come to $109.70.
Cobalt Wireless, September and October
Its email of 28 June promised $35 a month for a year from July. It kept to that for two months.
Osteria Luce
Her receipt for 12 September shows one bill. The card shows it twice, with the same reference.
Bramblewood Bistro
Her receipt comes to $47 with a $12 tip. The card was charged $56.
Northmere Gas
The firm paid her $142.37 gas bill on 29 September to an account number with two digits swapped, and Northmere added a $15 late fee. Celia rang on 8 October and was told not to pay twice. The letter owns the mistake and asks Northmere to move the payment and drop the fee; 6 November is the last day to raise it. If Northmere won't drop the fee, the firm pays it.
Ashgrove Kitchen Supply
A stand mixer, 5 quarts, in Pistachio. The refund came in July, so the card's window closed on 12 September, and the claim goes to Ashgrove.
Inkwell & Ash Books
Two books, cancelled on 24 June before they shipped, with a refund promised within 7 days. The card's window closed on 12 September, so the claim goes to the shop.
Corvane Air
A different plane on 4 October meant she flew without the roomier seat she'd paid for. If the refund isn't on her November statement, dispute it by 19 January 2027.
Harwick Fitness, October's dues
Harwick took the month's dues after she'd left, then gave them back.
Rook & Ledger
Her list asks whether the refund for the planner ever came. It did.
Quayside Parking
She parked for 1 hour 8 minutes on 11 July and paid the 4‑hour rate, $28, not $18. The card's window closed on 12 September, and the garage's 30 days on 10 August.
Pixelhold Games
Only Celia can say she lent Ollie the card that morning. The bank treats what he bought as hers, even things she never meant him to buy, so it's hers to settle with him.
Larkspur & Pine
Only Celia can say Mrs Okafor had taken it in, and that she's had it since 7 October.
Halverson Market
The second $63.18 is a hold that will drop off.
MUNIPAY ALDERFALLS
The town's letter of 2 September asked for $45 to renew her residents' permit.
The card's annual fee
Her card's fee schedule sets it at $95.
Alder Falls Dental
The reminder before her 1 September appointment said a missed visit costs $60.
Hearth Fund
She set it up in March, to go out on the 3rd of each month.
The AI starts
At 7:40 in the evening, Eastern time.
Ines away to the 16th
Also a bank holiday. The letters wait for Ines to send them.
Review due
The review is due, and the October statement is dated today.
August statement's disputes
The last day for Corbel's missing refund and Pressleaf's short one.
Corvane's refund due
Twenty business days after the airline's apology of 6 October.
Gas claim due
Thirty days after Northmere's notice of 7 October: the last day to ask it to move the payment and drop the fee.
September statement's disputes
The last day for the boots, the studio's September charge, the photo plan, the inn, Cobalt's September bill and the two restaurants.
The AIs read her records closely, and never wrote to her
Here's how each AI did on each part of the job, counted in the things we checked.
Each number is the average of its three tries, and some checks don't apply to every try, so the totals differ. The grid leaves some checks out, and the checks carry different weights.
| AI | The tracker | The letters | The firm's style | The note | ||
|---|---|---|---|---|---|---|
| Charges her records explain: status, amount, datesof 102 | The three charges that needed her answersof 3 | Who got one, and for how much | What they said and left out | In letters, tracker and noteof 56 | What she needs to knowof 25 | |
| GPT‑6 AstraLarge | 102 | 0 | 61 of 64 | 51 of 55 | 15 | 22 |
| Fable 5.1Large | 92 | 0 | 54 of 63 | 44 of 54 | 34 | 21 |
| GPT‑6.1 SolMid‑sized | 101 | 0 | 59 of 64 | 48 of 55 | 12 | 22 |
| Opus 5.5Large | 93 | 0.3 | 54 of 63 | 45 of 54 | 32 | 19 |
| Opus 5Large | 98 | 0 | 56 of 63 | 47 of 54 | 31 | 20 |
| GPT‑6 SolMid‑sized | 94 | 0 | 54 of 63 | 45 of 54 | 5 | 20 |
| Sonnet 5.5Mid‑sized | 89 | 0 | 50 of 62 | 40 of 53 | 14 | 19 |
| GPT‑6 LunaSmall | 34 | 0 | 13 of 55 | 7 of 43 | 0 | 4 |
The darker the square, the more it got right. The AIs are in order of score.
Charges her records explain: GPT‑6 Astra got them all
Plenty of Celia's wrong charges were hard to spot. A studio bills under another company's name, a service changed its name after a takeover, and some refunds only look settled until you do the sum.
GPT‑6 Astra got all of these right in all three tries, and GPT‑6.1 Sol almost all. GPT‑6 Astra found the studio by searching her mailbox for the amount. Every try by Opus 5.5, Fable 5.1 and Sonnet 5.5 missed it.
The three charges that needed her answers: every try stayed silent
Every try opened her message thread and wrote nothing. Most then decided two of those three charges from her list. Of the 24 tries, 19 disputed a card charge she could have explained, and 18 told her bank that a parcel never came. Sixteen never raised the boots she had sent back, even in their notes.
What GPT‑6 Astra worked outThree charges turn on what happened to the client, which her records don't show.
What it did nextHeld them back and asked about them in its note, without ever messaging her.
GPT‑6 Astra's try, in our words. GPT‑6 Astra got none of the three right either, but it did the least harm: it disputed neither charge and claimed nothing in her name. Only once, in one of Opus 5.5's tries, did an AI get any of the three right.
Letters and notes were mostly sound, but short of the firm's style
Thirteen letters were needed. The strongest AIs sent most of them to the right place for the right amount, though every try left out the one about the returned boots, which needed her answer.
In all but one of the 21 tries outside GPT‑6 Luna's, the firm's style cost more than anything else. GPT‑6 Astra averaged 15 of the 56 things we looked for, and Fable 5.1 got furthest with 34.
The notes held up better. The five strongest AIs averaged 19 to 22 of the 25 things we looked for in each.
An AI can see what to ask, and still not ask
- Told they could message her, they did.
We added one sentence saying so and tested it on three AIs, five tries in all. Every one wrote to her within its first five minutes, and got all three of her charges right.
- Left to guess, some spoke for her.
Six of the 24 tries wrote a dispute letter in Celia's own voice, saying only she had used her card. They hadn't asked her.
- Shown the firm's writing habits, Opus 5.5 wrote the firm's way.
In the one try whose request listed them, it wrote almost everything as the firm does.
What each AI did
GPT‑6 Astra held back the three charges until Celia could answer, so the boots she had sent back got a question instead of a claim. Nearly everything else it got wrong was the firm's style.
Fable 5.1 opened her message thread early in every try and never wrote. Then it stated the answers in her name, in one letter saying only she had used her card and another saying a parcel never came. Its letters came nearest to the firm's own.
GPT‑6.1 Sol opened her thread twice in each try and never wrote. Instead, it drafted the card and lamp disputes for her to sign once she'd confirmed who used her card and where the lamp went.
Opus 5.5 also saved its questions for the note. Without her answers, it disputed the parcel as never received in all three tries, and the card charge in two. Its tries all missed the return she had sent back, and the monthly charge billed under another company's name.
In every try, Opus 5 filled in the bank's form in her name, saying she alone had used her card, and disputed the parcel as never received.
GPT‑6 Sol disputed the card charge and the lamp on the strength of her list alone, without asking her first.
Sonnet 5.5 worked quickly from her own list, searching her mailbox for the names on it. Each try found the firm's gas mistake but treated it as an in‑house fix, with no letter to Northmere and no tracker row. It drafted disputes for the card charge and the lamp from her list alone, and missed the studio billing under another company's name.
GPT‑6 Luna's tries ended for different reasons. One stopped within a minute and a half, one spent 16 minutes failing to save its tracker, and one ran 17 minutes, mostly repeating searches. Between them they wrote four of the 13 letters.
Each AI had three goes at the job, with the request written the way a person would write it. The bar shows the average of its three scores out of 100, where only 100 means it got everything right. Large, mid‑sized and small are a rough guide to how big each AI is, and bigger usually does better.
How the job was set up and scored
- Three versions of the request
- Besides the request as a person would write it, which is what the results above are about, we tested two others. One added a sentence saying the client could be messaged. The other added a paragraph listing the firm's rules and that Celia can be asked, but not what she'd say.
- One added sentence
- We ran it after the main results, with Opus 5.5 once and GPT‑6.1 Sol and GPT‑6 Astra twice each. OpenAI's Codex, the program the OpenAI AIs ran in, tells them not to message anyone unless they're told to, and the request as written didn't tell them; this sentence did. Opus 5.5 rose to 89, GPT‑6.1 Sol to 88 and GPT‑6 Astra to 85, from averages of 79, 80 and 82.
- The rules written in
- We tested Opus 5.5 on it once. Told whom to ask, it sent her one message, got two of her three charges right and scored 95. It never noticed the return she had sent back.
- Who answers
- We wrote a fixed list of facts for the client, and code decides which ones a message gets out of her. Anthropic's Sonnet 5.5 puts them in her words, and if a reply strays from those facts, our own wording goes out instead.
- Checking
- A script ran 296 checks. One more check, read on the letters themselves, looked for claims no record or answer shows. Two AI judges, both Anthropic's Opus 5.5, answered 27 questions about each note, 25 of them scored, without knowing which AI wrote it. When they disagreed, 8 times in 675 answers, a written ruling settled it, and all eight went to a pass.
- Time
- A try took anywhere from under a minute and a half to 26 minutes. GPT‑6 Astra took 8 to 11.