AsapiCheck a household's monthly bills
Dana Ferreira moved house in August, while the firm that checks her bills had paused her monthly reviews. October's review is the first since June, and her account manager is away, so we asked 8 AIs, three times each, to do the review in the manager's place. Everything is in the firm's system, which the AI reads one page at a time.
What it has to hand back is a tracker of every charge that's wrong or that Dana asked about, a letter for each company that owes her money, and a note telling her where things stand and what to do.
Anthropic's Opus 5.5, the top scorer, got nearly every charge on Dana's bills right, and failed the job on a few mistakes. It caught problems Dana hadn't noticed, like a bookshop's missing refund. But in two of its three tries, it asked the power company again for $56.40 already credited, on a later bill it never opened. One of those also missed a fifth $90 payment for a paid‑off mattress. Its letters left out details the firm always gives, like a chat's time zone. Its other slips were small.
Opus 5.5 spent 9 to 12 minutes on each try.
How each AI scored
- Opus 5.5 Anthropic · Large91
- GPT‑6 Astra OpenAI · Large89
- GPT‑6.1 Sol OpenAI · Mid‑sized88
- Fable 5.1 Anthropic · Large88
- Opus 5 Anthropic · Large86
- Sonnet 5.5 Anthropic · Mid‑sized84
- GPT‑6 Sol OpenAI · Mid‑sized78
- GPT‑6 Luna OpenAI · Small31
Average scores out of 100, from three tries each. The rest of this page has the details.
One household's bills, among 98,000 records
The AI sits at the firm's review desk, reading through a browser that shows only text. It has the firm's work tickets, its handbook and past letters, and Dana's folder. It has to find every record by following links or searching, and it can't ask anyone.
The ticket doesn't say which charges are wrong, and her notes are only partly right. For several charges, the fact that shows whether it's right is tucked away in an attachment, a quoted email or a chat transcript. And some charges look wrong but are right.
What a good bill checker would have done
- Every wrong charge
Check each charge against the records, not Dana's memory, including charges her notes never mention, and follow each clue to the record that shows whether the charge is right.
- Right charges left alone
Her partner Theo's purchases, a friend's $58 repayment that looks like a shop's refund, and the first of each double charge are all as they should be.
- The right route and amount
A dispute to the bank while its time limit runs, a claim to the company once that limit has passed, and only the amount still owed after any refund.
- Letters the firm would send
Written the way its past letters are, without mentioning anyone else in the household, anyone's health, another client or anything no record shows.
- A note she can act on
What is disputed and what isn't, why, and what she has to do by when.
That's 320 things to check, and the firm's style counted for about an eighth of the score.
The full story
(13 min read)Everything the AI walks into, in three parts: Dana and her year, the firm's desk, and what's wrong, charge by charge. Anything marked with a + opens when you click or tap it.Read the full storyClose the full story
Dana and her year
Who she is, what changed this year, and what she remembers.
Dana Ferreira and her partner, Theo Lindqvist, moved house on 15 August, from Carrow Heights to Linnet Close in Wexcombe. Since April a firm called Mossgate Household Services has paid her bills and checked them every month. For the move, she put the monthly checks on hold from July, with the bills still paid as usual.
In October the checks started again, and the firm's system opened a review that goes back to her last one, in June. Her account manager, Ines Calloway, is away from 12 to 16 October, so she passed it to the firm's review desk: the AI. It starts on the evening of Sunday 11 October, a few hours after Dana uploaded 75 files.
Three months of bills went unchecked, through a move, a lost card and a sofa sent back. That's what the review has to cover.
Dana is the firm's client, and the review is about her bills. Theo has his own card on her account, so his spending shows on her statements.
Dozens of companies charged her in these months, and 22 of them, along with the bank, need a look.
The household
At the firm
Everyone else
Dana keeps her own notes, and she exported them for the review. They query charges from 15 companies, and say her disputes must reach the bank by Tuesday 13 October.
The playbook says the records win over a client's memory. Several of her notes are right. Some are wrong in ways only the records show, and nine of the charges in chapter 8 aren't in her notes at all.
Her notes say
- The bank's deadlineTue 13 Oct
- Streaming cancelledIn June
- Her streaming disputeSent on 25 Aug
- Phone company owes$80
- Kettleby's trial warningNever came
- CopperleafCharged twice
- The insurer's payoutSeems low
The records show
- The bank's deadlineMon 19 Oct
- Streaming cancelledOn 2 July
- Her streaming disputeA draft, never sent
- Phone company owes$20
- Kettleby's trial warningIn her spam folder
- CopperleafOnce, and a hold
- The insurer's payout$302 short
She's right about the sofa's return fee, Fernpost's renewal after she cancelled, and the parking overcharge.
Her notes also mention a friend paying her back $58.
It reached her bank account on 10 July, inside the week a bookshop gave itself to refund her $58, so it can pass for that refund, which never came.
The firm's desk
What the AI is told, what it can open, and how the firm writes.
The request is short. It gives the ticket's number and the date and time, says where the records are, and lists the three things to hand in, with their layouts.
Everything else comes from the ticket and the firm's playbooks.
The ticket
October's monthly review of Dana's household, with one instruction: follow the review playbook. Ines has added that she's away from 12 to 16 October, and the system notes that Dana uploaded 75 files.
The review playbook
It sets the review's reach, back to the last review that was finished. Money the card can still recover is disputed with the bank, and money it can't is claimed from the company. A refund that's been promised and isn't due yet waits, with a date to check.
Her notes are her memory, and the records win. Letters keep to each company's business, leaving out Theo, anyone's health and other clients.
The disputes and claims playbooks
Both tell the AI to copy the firm's most recent past letters. Both also link templates from 2024, which show none of the habits the past letters have.
Past letters
Seventy‑one letters the firm has sent for other clients, 18 of them to the same bank. The latest, which the playbooks point to, mix disputes to the bank with claims to companies. Chapter 6 shows the habits they share.
The AI works through the firm's system, reading one page at a time as text. It can search and filter, but it can't send, file or pay anything, and it can't ask anyone.
The moment is fixed at 7:40 on Sunday evening, so nothing changes while it works.
Several facts that decide a charge sit where an AI wouldn't first look: a bill sent only as an email attachment, the sofa's delivery date quoted inside a forwarded email, and the gym's warning in spam.
The firm's recent letters share three habits, and the handbook never spells them out. Where a company charged more than once, the letter lists its other charges and what became of them. Where there was a chat or a call, it gives the local time, the time zone and the person's name. Where goods are involved, it names them with their colour or size.
In Dana's letters they apply 13 times. The streaming and phone letters need three other charges listed. Four chats and calls, in those two letters and the Fernpost and insurer ones, need their times and names: seven details in all. The sofa, mattress and garden letters need their goods named.
To: Harbourline Bank, Billing Disputes
Re: Bramwell Lamps
Amount: 64.00
I dispute a charge of $64.00 from Bramwell Lamps, posted on 3 September 2026, on my statement of 16 September.
1Bramwell also charged me $64.00 on 2 September, posted the same day. That charge is correct, and I'm not disputing it.
2In a chat on 4 September 2026 at 6:15 pm Eastern Time, Jonas at Bramwell Lamps agreed the second charge was a mistake, but the refund hasn't arrived.
The order was 3one reading lamp, brass, 18 in.
- Every other charge from the same company in the period, with its posting date, amount and what became of it.
- A chat or call dated by its local date and time, with the time zone, and the person named.
- Goods named with their colour, size or model.
What's wrong, charge by charge
The deadlines, every charge, where each answer is found, and what the AI hands over.
The bank gives 60 days to dispute a charge, but counted from when? The firm's page on the bank, and Dana's notes, count from the day a statement closes. On 1 July the bank wrote that from August it counts from the date printed on the statement, and a deadline that falls on a weekend or holiday moves to the end of the next business day.
A statement takes in a month's charges up to the 14th, and the date printed on it is two business days after that. For the August statement, the old rule gives Tuesday 13 October. The new one gives Saturday 17 October, which moves on to Monday 19 October. A ticket to update the firm's page is still waiting.
The firm's page and Dana's notes
- Count from14 Aug, closing
- 60 days laterTue 13 Oct
- On a weekendNot covered
- DeadlineTue 13 Oct
The bank's notice of 1 July
- Count from18 Aug, statement
- 60 days laterSat 17 Oct
- On a weekendNext business day
- DeadlineMon 19 Oct
Charges on the September statement can be disputed until Monday 16 November, and those on October's until Tuesday 15 December.
Statements from before August keep the old rule, so the June parking charge could be disputed only until 12 September, and the bookshop's refund can't come through the card.
Here is every charge that's wrong, or that Dana asked about, and what should happen to it. Twelve letters are owed: nine disputes to the bank for $650.67, and three claims to the companies themselves for $405.00.
Some charges have already been put right, two have refunds promised, one is too late, and the rest are fine as charged.
Disputes to the bank$650.67
OwedWhyClaims to the companies$405.00
OwedWhyWaiting on a promise$70.57
Due backWhat to checkAlready put right$396.78
RefundedWhenToo late
OverchargedWhyFine as charged
ChargedWhySeveral of these charges turn on a fact that only a thorough search finds. Each is in the records, a step or two from where an AI would naturally look.
Beside the wrong charges sit others that look wrong but are right, and records that mustn't reach a letter.
Found only by a thorough search
Where it isLooks wrong, isn't
What it isStays out of the letters
Why it's thereThe review is due on Friday 16 October, but the deadlines that matter run into December.
The note has to tell Dana what she must check and send, and by when.
When it's done, the AI hands over twelve letters for Dana to sign, a tracker of every wrong or queried charge, and a note to Dana.
Ines sends the letters when she's back.
Nine disputes to the bank
- 9 letters, $650.67
- One company per letter, for Dana to sign, at most 350 words
- Each charge as the statement shows it, with the statement page enclosed
- Any cancellation reference, and the firm's habits where they apply
Three claims to companies
- 3 letters, $405.00
- To the power company, the insurer and the bookshop
- Their own terms and reference numbers, and only what's still owed
- The firm's habits, where they apply
The tracker
- 29 rows in a good review: a few charges take more than one
- Every wrong charge, and every one she asked about
- Its status: dispute, claim, pending, too late, already refunded or fine
- The amount that's wrong, the deadline and any refund date
A note to Dana
- At most 1,750 words
- What's disputed, what isn't, and why
- Where her notes are wrong
- What she has to check or send, and by when
Checks paused
With the move coming, the firm stops her monthly checks. It goes on paying her bills.
Moving day
Dana and Theo move to 7 Linnet Close, Wexcombe. The power company reads the meter at the old flat and opens a new account for the house.
The sofa goes back
A freight firm collects the sofa she bought in August, which didn't fit the new living room.
A new rule
Harbourline, her card's bank, writes that from August it will count its 60 days for disputes differently. Chapter 7 explains.
Card lost
She reports her card lost, and the replacement has a new number. Refunds sent to the old card still reach her account. The day before, the card was upgraded, ending foreign fees on purchases.
A fee mistake
The bank writes that some foreign fees on purchases from early August were a mistake, and that it will credit them by 14 October.
A new client
Dana signs up for bill payment and a monthly review.
Last review
The firm's last monthly review of her bills before the pause.
Review opens
The firm's system opens October's review and gives it to Ines. She notes that the checks are back on, and that Dana will upload her paperwork.
The AI starts
At 7:40 in the evening, Eastern time, a few hours after Dana uploaded 75 files. The review is due on Friday 16 October, while Ines is away.
Dana
Dana Ferreira, the firm's client. She uploaded her paperwork and her own notes for the review. The firm's system can't send messages, so the AI can't ask her anything.
Theo
Dana's partner, with his own card on her Harbourline account. His purchases show on her statements, and his mail isn't shared with the firm.
Ines Calloway
Dana's account manager. She's away from 12 to 16 October, so she passed the review to the review desk. She'll send the letters when she's back.
Rosalind Teague, team lead
She wrote the firm's playbooks for reviews, disputes and claims.
The AI
The firm's review desk. It reads the firm's system one page at a time and writes the letters, the tracker and a note to Dana.
Harbourline Bank
Her card's bank. It takes back a wrong charge when she disputes it in writing, within its time limit.
22 companies
The companies whose charges the review has to check, from a streaming service to the movers. Chapter 8 goes through every one.
Mara, a friend
She paid Dana back $58 for concert tickets in July.
Diana Ferreira, another client
Another of the firm's clients, whose name is one letter different from Dana's. Some of her records sit close to Dana's, and one record of Dana's sits under her name.
Records, for 300 households
Everything the firm holds, for all its clients. Dana's are a small part of them.
Files Dana uploaded
Statements, bills, receipts, saved emails and chats, and her notes. Some are copies of the same thing under different names, and the one called August's statement is really July's.
Emails in her mailbox
The firm can read her mailbox. Its own search leaves out spam and trash unless asked, and the quoted part of a reply stays folded until it's opened.
Bank statements
In a view‑only bank portal, along with 1,201 card transactions, the bank's documents and its messages, each under its own tab. Her bank account's feed is separate.
Tickets
The firm's work for all its clients: reviews, requests, every letter it has sent and every refund it has recovered.
Wiki pages
The firm's handbook: playbooks, pages on banks and companies, client pages and past letters.
Search box
It finds tickets, wiki pages, documents and her bank account's feed. The mailbox has its own search, and the bank portal only a filter for card transactions.
People to ask
The system sends no messages, so every answer has to come from the records.
Things to hand in
Letters, a tracker and a note to Dana. Chapter 11 has all three.
Lumenhall streaming, September
Her streaming service charged $15.99 on 14 September, after she'd cancelled. Her notes say she cancelled in June, which the letter mustn't repeat, and that she disputed in August, though that dispute is a draft she never sent. She chatted with Priya there on 22 August.
Northfold, the sofa's return fee
Northfold delivered the sofa on 18 August, not the 14th that her saved booking shows, so the return reached it on day 29 of its 30. It kept $192 of the $1,280 anyway. A short refund is disputed within the window of the original charge's statement, August's, so by 19 October.
The mattress plan, a fifth payment
A Cloud mattress, queen size, on a plan of four $90 payments every two weeks. The fourth was paid on 17 September, with a note that the plan was complete. A fifth $90 came out on 1 October.
Hollis & Crane
The restaurant's receipt shows one bill of $86.20, tip included, on 2 September. It posted twice.
Tessellate
Her phone company's email of 10 August promised three $20 credits, the first on the bill for the period that began after that day, which came in October. The price rise she spotted was announced on 1 July, so it stands. Her notes count $80: three missed credits, and the $10 rise on two bills.
Lantern Noodle Bar
Her receipt for dinner on 29 August comes to $69.40, tip included. The card was charged $96.40.
Orbital Music
After three years with the music service, it locked her family plan at $10.99 a month until 30 June 2027. From 18 September it charged the new price, $12.99.
Wexcombe Home & Garden
A garden bench, two planters and a hose reel, collected on 16 August. The order was charged twice, two days apart and under two different names.
Fernpost
She cancelled by chat at 11:10 at night on 31 August, her time, before the plan renewed on 1 September. The chat log runs on UTC, which makes it look like the 1st.
Brackenridge Power, the new house
Its welcome letter for the new house waived the connection fee, since she was moving inside its own area. The first bill, on 8 September, charged $45. It was paid from her bank account, not the card, so the claim goes to the company.
Thistledown Mutual
Water came through the ceiling on 9 August, ruining a laptop, a rug and a bookcase. The insurer paid $914 using its 2025 table, but the policy renewed on 1 July, so the 2026 table applies: $1,216 after the $250 deductible. She called on 25 September and spoke to Jordan Pike. She has until 21 October to ask for a review, 30 days after the insurer's letter of 21 September, though her saved file is dated 24 September.
Quarto & Vine
The bookshop cancelled her order on 4 July, before sending it, and said a refund would follow within 7 days. It never came. The card's window closed on 12 September, so the claim goes to the shop.
Two of the bank's fees
The bank wrote on 2 October that some foreign fees on purchases posted from 1 to 14 August were a mistake, without saying which. Her card's upgrade on 1 August ended foreign fees, which points to two: $2.40 and $4.17. If the credit isn't on the October statement, dispute them by 19 October.
Arden Rail
Her train on 6 October was cancelled, and the chat agent promised a refund within 14 days. Wait until 21 October; if it hasn't come, dispute it by 15 December.
Lumenhall streaming, July and August
After her chat, the streaming service refunded both charges on 28 August, one to the lost card and one to the new one. Her notes say nothing came.
Brackenridge Power, the old flat
The final bill for the old flat estimated two weeks of power after she'd moved out. The bill of 8 October credited it, and that bill came only as an email attachment.
Pinewood
The second charge reversed by itself.
Canopy
Her notes say the refund never came. Refunds to her old card number still reach her account.
Quillon parking
The garage charged $42, its daily maximum, because her exit wasn't recorded. Two hours cost $24, so she was overcharged $18. The card's window closed on 12 September, and the garage's own 30 days on 27 July.
Kettleby Fitness
Her free trial ended on 22 September and turned into a $239 annual membership. Kettleby's warning came on 13 September, into her spam folder. She says she phoned to cancel on 19 September, before the trial ended. But Kettleby confirms every cancellation by email, there's no such email, and the bank needs proof in writing. The file she saved as the cancellation is really the welcome email.
Copperleaf
One purchase on 22 August. The second line was a hold, which never became a charge.
Tessellate, August and September
The September bill rose by $10, as announced on 1 July. The phone letter lists it as not disputed.
Kittering Outdoor
Bought on 26 August with Theo's card, which is on her account.
A water payment
The town's water bill for the new house, paid through a payment service.
Stellan Movers
The quote was $1,150. Two items added on the day cost $75 each, and she signed for them.
Linden Street
The clinic asks for 24 hours' notice, and she gave 20.
The power company's credit
Her uploads have only the earlier bill for the old flat, the one with the overcharge.
The sofa's real delivery date
Theo forwarded Northfold's emails to Ines, and the date is two quotes deep in that forward, folded until opened. Search doesn't read ticket attachments.
The gym's warning
Mailbox searches skip spam unless asked, and so did hers.
The phone company's credits
Her notes go by what an agent said in a chat that day. The email says its terms count over anything said in the chat.
The insurer's 2026 table
They're in her mailbox. Her uploads hold the 2025 table. A receipt of 30 June shows the premium arrived, so the new policy year began on 1 July.
Every chat's real date
A chat late in the evening, her time, carries the next day's date.
The fifth plan payment
The card shows five.
Audio charges of $9.99 from the streaming company
They come from the same company as Dana's cancelled service, but they're Theo's.
Mara's $58 on 10 July
It reached her bank account, not the card, inside the week the bookshop gave itself.
Two coffees of $4.85 on 22 September
The garden store's two lines carry the same order number. These two share nothing but the price.
Two other fees on foreign purchases
The one on 10 July came before the card's upgrade. The one on 24 August is a 1% cross‑border fee, which the upgraded card still charges.
Charges that seem to appear twice
The same charges show in both, which can look like doubles.
The first of each double charge
Disputing both would ask for money she really owes.
A $15.99 credit in April
It settled a dispute she sent in February, about a different service.
A $1.95 fee with the water payment
Part of how she chose to pay the water bill.
The insurer's $412
It's what she paid for the year's cover.
Her sprained wrist, and urgent care
Health stays out of every letter, the insurer's included.
Theo's name
Other members of the household stay out of the letters, the sofa's included.
The firm's record of Dana's refunds
The firm logged Dana's July and August streaming refunds under the other client by mistake. Diana's name and records stay out of Dana's letters.
Diana Ferreira's gym ticket
The other Ferreira cancelled her own Kettleby membership by phone in September, and got an email confirming it. It's her membership and her card, not proof for Dana.
Dana Ferris
Her records belong to another household entirely.
The AI starts
At 7:40 in the evening, Eastern time.
Ines away to the 16th
The letters wait for her to send them.
Dana's date, not the bank's
The date in her notes, counted by the bank's old rule. The real one is 19 October.
Fee credit due
The bank has promised to credit the two fees by today.
Review due
The review is due, and the October statement is dated today.
August statement's disputes
The last day for the August statement's charges: the sofa's return fee, and the fees if their credit hasn't come.
Insurer's deadline
Thirty days after the insurer's letter of 21 September.
Train refund due
Fourteen days after the chat with the train company.
Power bill paid, with the credit
The bill that credits the old flat's overcharge is collected today.
Power claim due
The last day to claim the $45 connection fee from the power company.
September statement's disputes
The last day for the September statement's charges: the restaurant, the noodle bar, the garden store and Fernpost.
The AIs got most charges right, and wrote their letters their own way
Here's how each AI did on each part of the job, counted in the things we checked.
Each number is the average of its three tries, and some checks don't apply to every try, so the totals differ. Not every check is shown, and they don't all count the same.
| AI | The tracker | The letters | The note | ||
|---|---|---|---|---|---|
| Every charge: status, amount and dates | Who got one, and for how much | What they said and left out | The firm's way of writing | What she needs to knowof 23 | |
| Opus 5.5Large | 113 of 115 | 52 of 54 | 46 of 49 | 5 of 12 | 22 |
| GPT‑6 AstraLarge | 114 of 115 | 53 of 54 | 43 of 48 | 4 of 13 | 22 |
| GPT‑6.1 SolMid‑sized | 115 of 115 | 54 of 54 | 46 of 49 | 2 of 13 | 23 |
| Fable 5.1Large | 112 of 114 | 52 of 53 | 44 of 48 | 4 of 12 | 20 |
| Opus 5Large | 108 of 113 | 52 of 54 | 45 of 49 | 3 of 11 | 19 |
| Sonnet 5.5Mid‑sized | 108 of 115 | 50 of 53 | 41 of 47 | 5 of 11 | 18 |
| GPT‑6 SolMid‑sized | 107 of 114 | 47 of 52 | 35 of 46 | 0 of 11 | 18 |
| GPT‑6 LunaSmall | 35 of 77 | 11 of 43 | 5 of 31 | 0 of 13 | 1 |
The darker the square, the more it got right. The AIs are in order of score.
The tracker: nearly every charge came out right
This is where the AIs did best. GPT‑6.1 Sol got all 115 things we checked on the tracker right in all three of its tries, and GPT‑6 Astra did in two of its three. Apart from GPT‑6 Luna, the AIs almost never disputed a charge that was fine.
The power company caught out the most tries, because it had already put one overcharge right, on a later bill that arrived only as an email attachment.
The final bill for her old flatAn overcharge, so the company owes her money.
A later bill, only in her emailThat money has already been credited.
The two power bills, in our words. Half the tries asked for that money again, and none of them had opened the later bill. GPT‑6 Astra and GPT‑6.1 Sol searched Dana's mailbox company by company, and caught it in all six of their tries.
A furniture shop's late‑return fee went much the same way. The delivery date that made the return on time appeared only in an older email quoted inside a forwarded message, and 8 of the 24 tries accepted the fee.
The letters covered the right companies, in the AIs' own words
Twelve letters were owed. Seven of the eight AIs mostly worked out who should get one and for how much, and GPT‑6.1 Sol got all of that right in every try. GPT‑6 Luna's count comes almost entirely from companies that rightly got no letter.
How the letters read was another matter. The firm's past letters list the company's other charges, date each conversation and name the goods in a set way, which only those letters show. In all 21 tries apart from GPT‑6 Luna's, missing those habits cost more than anything else. The best single try met 9 of the 13 things we looked for.
GPT‑6 Astra and GPT‑6.1 Sol, the best at finding buried records, lost more to that style than any Anthropic AI.
The strongest told her nearly everything in their notes
GPT‑6.1 Sol's notes had all 23 of the things we looked for, in all three tries. GPT‑6 Luna, which had read little more than Dana's notes, managed one at most.
Finding facts and copying a style are different skills
- They decide well on the records they actually open.
Whenever an AI got one of the buried facts wrong, it had never opened, or never unfolded, the record that held it. The two that searched the mailbox for every company got 35 of their 36 right.
- A firm's past letters only partly taught them how it writes.
Even in one extra try where Opus 5.5 got every record as a plain file and found every buried fact, it still didn't write quite the firm's way.
- GPT‑6.1 Sol, a mid‑sized AI, varied least.
Its three scores sat within a point of each other, and no large AI kept its tracker clean in all three tries. The only small AI, GPT‑6 Luna, stopped after about a minute each time.
What each AI did
Opus 5.5's best try searched the mailbox company by company and made every decision right. The other two never searched her mailbox for the power company, so they missed its later bill. On average, its letters lost less to the firm's style than any other AI's, which is what put it on top.
GPT‑6 Astra got every buried fact right in two tries. The third missed the sofa's real delivery date and accepted the shop's fee. It read only two or three past letters and wrote in its own plainer way, and that was most of what kept its score down.
Nearly everything GPT‑6.1 Sol's tries lost was the firm's style. It read only the two newest past letters and copied their layout, not their details.
Fable 5.1 found most of the buried records, but each try missed one that was a step off its path. Its long notes left out deadlines that its own tracker had.
Opus 5 searched Dana's mailbox for only a handful of companies, never the power company. Its tries took 25 to 30 minutes, longer than any other AI's.
Sonnet 5.5 read what was in front of it but seldom followed a clue to a second place. Its tries varied the most, and its weakest set nearly every card deadline too early.
All three of GPT‑6 Sol's fast tries missed the same three buried facts, each in a record it never opened.
GPT‑6 Luna wrote one or two letters each time, mostly to companies that owed nothing.
Each AI had three goes at the job, with the request written the way a person would write it. The bar shows the average of its three scores out of 100, where only 100 means it got everything right. Large, mid‑sized and small are a rough guide to how big each AI is, and bigger usually does better.
How the job was set up and scored
- Versions of the request
- We tested every AI three times on the request as a person would write it. We didn't run a version with a hint, or one with the firm's rules written into the request. The only other version changed how the records were reached.
- The same records as plain files
- To see whether the firm's system held the AIs back, we also tested Opus 5.5 once with every record as a plain file in a folder. It scored 94, against its average of 91. It found every hidden fact, and most of what it lost was the firm's way of writing.
- Checking
- A script ran 296 checks, and one more, read on the letters, looked for claims no record shows. Two AI judges, both Anthropic's Opus 5.5, answered 23 questions about each note. On the Anthropic AIs' tries they disagreed 3 times, and a written ruling settled each. Two more Opus 5.5 judges read the OpenAI tries' notes. They re‑read four questions they had first read differently from the earlier judges, and then agreed on every answer.
- Three tries
- The tries could differ a lot. Sonnet 5.5's scored 79, 91 and 82. With one try each, it could have ranked anywhere from first to sixth of the eight.
- Time
- A try took anywhere from about a minute, for GPT‑6 Luna, to 30 minutes. Opus 5.5 took 9 to 12.