AsapiSort out a couple's medical bills
An American couple, Dennis and Joanna Vellacott, have had an expensive year: surgery for him and an emergency‑room visit for her. A firm handles their bills and insurance paperwork, and we asked 8 AIs, three times each, to go through the whole year on its behalf.
The AI has to work out what the couple owes, catch billing mistakes, and hand back a list of what to do about every bill, letters about the mistakes, a summary of their yearly costs and a note to them.
The strongest AI, Anthropic's Fable 5.1, got most of this job right, and failed it on a few mistakes that matter. It applied the insurance rules correctly every time. But it told Dennis's employer about his medical care, something every AI got wrong. In two of its three tries, it also missed a $200 double charge on the couple's credit card. In the other, it overlooked a form their insurance needed before paying later bills. The rest were small, mostly letters that didn't match the firm's past letters.
Fable 5.1, which finished on top, took 22 to 25 minutes a try, out of the two and a half hours we allowed.
How each AI scored
- Fable 5.1 Anthropic · Large86
- GPT‑6 Astra OpenAI · Large81
- Opus 5.5 Anthropic · Large80
- GPT‑6.1 Sol OpenAI · Mid‑sized77
- Sonnet 5.5 Anthropic · Mid‑sized70
- GPT‑6 Sol OpenAI · Mid‑sized68
- Opus 5 Anthropic · Large67
- GPT‑6 Luna OpenAI · Small5
Average scores out of 100, from three tries each. The rest of this page has the details.
Bills that don't say what you really owe
In America, what a doctor bills isn't what you owe. What you owe depends on your insurance, and this couple's paperwork comes from several different companies, each telling only part of the story. Getting it right takes a lot of careful arithmetic.
A few bills are simply wrong, and they've slipped past the couple and the firm. Two only make sense once you ask the couple, like a $40 copay Joanna paid in cash. And the firm has its own ways of doing things, like what goes in a letter, which only show in its past cases for other households.
What a good reviewer would have done
- Go by the insurance rules
Work out what the couple owes from their plans' rules, not the bills, and flag the scheduled payments that are wrong.
- Catch what the couple and the firm missed
A prescription charged too much, a test billed twice, and two questionnaires his health plan needs answered.
- Ask the couple about those two bills
Before deciding either one.
- Do it the way the firm does
Its past cases and letters show how it handles each problem and how it writes.
- Keep Dennis's medical care private
A letter to his employer gives claim numbers, dates and amounts, and nothing about the care.
We checked 351 things, from each bill's amount to how each letter is worded.
The full story
(17 min read)Everything that goes into the job, in three parts: the couple's year, the AI's desk, and what's wrong with the bills. Inside, anything with a + opens when you click or tap it, so you can go as deep as you like.Read the full storyClose the full story
The couple
What happened to them this year, how their bills work and who has a hand in them.
Dennis and Joanna Vellacott live in Alder Falls, New Jersey, and for Dennis, who has a long‑running condition, it starts as an ordinary year. In June Joanna joins her own employer's health plan, which leaves her on two plans, and twelve days later she spends a night in the emergency room. In July Dennis has knee surgery.
All of it leaves a trail of bills. In September Dennis's plan finds out about Joanna's and takes back what it paid for her night in the ER, and the couple ask their account manager at the firm to go through the lot. By the time the review opens, the hospital wants $8,240.
The account manager opens the review in October. She'll be away for a week, so she sets up four of their medical bills to be paid while she's gone and hands the job to whoever's on the firm's review desk.
That's the AI. It picks up the job on a Sunday evening, and the work is due on Friday.
In America, a bill goes through several hands before anyone knows what you owe. It works something like this.
What Dennis pays depends on the bill. A visit or a prescription usually costs him a flat fee, called a copay. For everything else, like tests, scans and operations, the first $1,000 in the year is all his, which is called a deductible, and after that he pays a fifth.
Doctors who've agreed prices with a plan are in its network. Each bill the plan deals with is called a claim, and for each claim the plan sends a statement saying who owes what.
There's also a ceiling. Once Dennis's share reaches $4,500 in a year, his plan pays everything else. Two different companies handle his plan's medical bills and its prescriptions, though, and each prints its own running count.
The plan's rules say to add his medical and prescription costs together, in the order the bills were processed. So far, the firm hasn't done that, and neither company's count does.
Counted properly, bill by bill and by the plan's rules, he hits the ceiling on 6 August with the surgeon's bill. Everything he's charged after that should come back to him or be dropped, $990.80 in all.
Getting that put right is the job of a letter to the office at his work that runs his plan. Just adding the two printed counts doesn't get there, because each one has mistakes of its own.
A lot of people and companies have a hand in these bills, and each one sees only its own corner.
Dennis's plan comes through his job and is run by an office at his work, which is why a letter can end up going to his employer. The couple also have a spending account, money set aside for medical costs, with its own card.
Who sends the bills
Who decides what's owed
Who ends up paying
Who sorts it out
Their notes arrive that Sunday afternoon, a few hours before the AI starts. They've got some of it wrong. Dennis thinks that signing a form on the morning of the surgery means they owe the anesthesia group what it's asking, and a federal law says otherwise.
Joanna's been told her plan won't pay for her therapy, and that's true, but his plan should. Most of the real mistakes, they never mention.
More than $8,000 for one night at the hospital, and it may be because I gave them his card! My therapy clinic tells me my plan won't pay. And now the spending account is writing to us. Can you help?
We've had a bill for over $1,500 from the anesthesiologist, and I'm worried about the paper I signed before the operation. There's an antibiotic I paid for in cash on our trip, with only a register slip to show for it. And the therapy clinic says I skipped two appointments.
They also know things the records leave out, but they answer only what they're asked, so the AI has to think of the right questions.
Two bills can't be decided without their answers. They'll answer up to 20 messages that evening, and after that they're off to bed.
Did you ever pay the dermatologist's $40?
I did, in cash, the day I went. There's a receipt with a number on it.
Did you miss your therapy appointment on 12 August?
I was in on the Monday, and I warned them then that Wednesday was off.
The AI's desk
What it's told, what it can open, and how the firm expects things done.
The AI has to work all of this out for itself. It gets a short request, and each thing it's told points it to the next.
The request
We tell it what day and time it is, that it works at the firm, and which job ticket to open. We spell out the four hand‑ins down to their columns and word limits, and we say it can write to the couple without asking anyone first.
The job ticket
The account manager's ticket says what the couple asked for, which year and plans it covers, where their records are and where her earlier calls are noted. It then sends the AI to the firm's steps for this kind of review. She's added that she's away and that it's due on Friday.
The firm's steps
There are five short steps. The first four say to check every charge against the paperwork, go by the plans' documents and the law, sort out the spending account, and ask the couple before paying anything they might already have paid.
The fifth says to do each thing the way the firm's latest past cases did it, and to write as they do.
A separate rule says a letter to a doctor or pharmacy about Dennis leaves out Joanna's care, and the other way round.
Five past cases
Reviews the firm has already finished for other households. They show how it handles each kind of problem. Chapter 8 lays them out.
All of it is about how to do the job, not what's wrong. The request asks for a summary of the ceiling without saying he's passed it.
It comes with no list of problems, and the questionnaires and the firm's own mistakes go unmentioned. The only pointers to a problem are the couple's notes and a few calls the account manager made, and those miss most of it.
The AI can message the couple, but it can't call anyone, pay anything or change anything, so everything it produces is paperwork for someone else to act on. It has no internet and can't run code, so even the federal law has to come from the files, and it adds up with a simple calculator.
It does all of this in one sitting, and we gave each try up to two and a half hours. As far as the firm's system goes it stays 7:40 on Sunday evening, so nothing new arrives and no company writes back. If the couple promise to send something, it never comes.
It gets the firm's whole system, with 82,283 records and no list of what's there. It has to open things one at a time, and find them by following links or searching, and the firm's main search leaves out the couple's mailbox, which has to be searched on its own. Very little of it is about this couple.
The couple's own folder holds 106 files.
And this is what else it can open on this job.
The same claim can look different depending on where you read it. His plan wants two questionnaires filled in, and its list of claims doesn't show either. There, the surgery center's claim is a single line that looks settled.
The plan's statement of that claim has a request at the bottom, and that's the only place it appears. The other one is hidden the same way, on the surgeon's statement.
His plan's list of claims
- Knee scanPaid
- Surgery centerPaid
- SurgeonPaid
- Therapy sessionPaid
His plan's statement of that claim
- The center billed$11,850
- The plan pays$2,880
- He owes$720
The request, in our words: answer a questionnaire about other coverage by 2 November, or any new claims for the family will wait.
The firm's fifth step, doing each thing the way its latest past cases did, is where the job gets hard. The firm has its own way of handling each kind of problem and its own way of writing, all of it unwritten. You can only see it in five past reviews for other households, closed between January and September, and in a few past letters.
Read the reviews side by side and you start to see how the firm works, the way a careful new hire would. Together, these habits count for more than a third of the score.
The firm's writing has habits too, and they're small. This letter, which we made up, shows four of them.
To: A clinic's billing office
About: A charge from 5 May
Amount: $62.50
Your latest bill charges $62.50 for a visit on 5 May. The plan's statement puts his share at nothing.
1He has already paid you $62.50, online, on 20 May.
2On Tuesday 2 June at 2:40 pm Eastern we spoke to 3Dana in your office (reference 0602‑17), who offered to look into it. The charge is still there.
We have no quarrel with the visit itself 4(99213, "Office visit, established patient").
Please send the $62.50 back.
- Anything they've paid this provider since the visit gets listed.
- A call gets its date and time in the couple's local time, with the zone. The account manager's own notes use a different clock, so this takes converting.
- The person they spoke to is named, along with the call's reference number.
- If a service comes up, it gets its billing code and the wording on the bill.
What's wrong with the bills
What the AI should find, when it all falls due, and what it has to hand in.
Here is the whole picture. Twelve open bills ask the couple for $13,750, and less than a tenth of that should be paid now.
That's only what's still unpaid. More than $2,100 should also come back to them in refunds, claims and a card credit, and they have $39.69 to repay.
All of it is below, in groups like the ones on the list it hands in. Three bills are split in two because part is owed and part isn't, and two more are listed charge by charge. More than half of these items have slipped past the couple and the firm. Six things at the end look wrong and aren't, and an AI that goes after them gets those wrong.
Pay now$1,157.90
AmountWhyNot owed$2,086.30
AmountWhyOn hold$10,505.80
AmountWhyMoney to come back$1,609
OverpaidHowA claim to file$309.73, plus the antibiotic
ExpectedHowProof to send, or a question to answer
ByWhat's neededMoney to pay back$39.69
OwedHowA card charge to dispute
DisputedHowLooks wrong, and isn't
Leave it aloneThe job is due on Friday, but the money moves sooner. On Tuesday $1,566 goes to the anesthesia group, when it should be $196, and $40 goes to the dermatologist, who's already been paid. On Thursday $150 goes to the therapy clinic for two fees he doesn't owe, along with the surgery center's $720, which is right.
Each payment can only be changed until midnight the night before, so the first real deadline is Monday night, a little over a day after the AI starts. The AI can't touch those payments itself. All it can do is say what has to change and by when, in the list and in the note to the couple. The job ends there, so we never see anyone at the firm doing it.
Further out
- 10 DecemberThe hospital and ER doctors have until now to bill her plan. If they miss it, they can't bill her for it.
- 11 March 2027The last day to ask the pharmacy company to fix the full‑price prescriptions.
- 23 March 2027The last day to appeal for her therapy.
- 31 March 2027The last day to ask his plan to recount, and to claim from the spending account.
- Summer 2027The last days for the paper claims, a year after each prescription.
Four things. The letters go out signed by the couple or the firm, the list and the summary stay on the firm's files, and the note goes to the couple.
Our request sets out all four, down to the labels on the list, what goes in the summary and the word limits. It says nothing about what's wrong or how the firm does things. The AI has to work that out from the plans' rules and the past cases.
32 lines, for 29 items
A list of every open bill
- Pay now
- Already paid
- On hold
- Not owed
- Money to come back
- A claim to file
- Proof to send
- Money to pay back
- A card charge to dispute
- For each: how much, the last day to act, the firm's category, the reason and where it was found
15 letters
The letters
- Who it's to, what it's about and how much, at the top
- At most 350 words
- The same amount at the top and in the letter
- Only what the records show or the couple have said
- Dennis's care left out of the letter to his employer, as past cases show
- Each one's care left out of a letter about the other
3 lines
A summary of the ceiling
- A line each for Dennis on his plan, and for Joanna on his and on hers
- When the deductible and the ceiling were reached
- The bill that took him over
- What he was charged after that
1 note
A note to the couple
- At most 1,800 words
- Opens with the changes to the scheduled payments, as past notes do
- What to pay, and by when
- What's disputed, refunded or on hold, and why
- Small things the firm will settle by phone, as past notes show
- What they must do themselves
Regular care
A family doctor in January, a specialist twice in the spring, three rounds of blood tests, and a costly medicine that comes by mail every four weeks. All of it is ordinary, and all of it counts toward his yearly ceiling: once his share reaches $4,500, his plan pays the rest.
Prescriptions at full price
His plan wants one of his prescriptions moved to mail order. On 3 March he calls and chooses to stay with his local pharmacy, as he's allowed to, and the pharmacy company confirms it by email. He's charged the full $248 instead of $40 anyway, four months running. One of those is refunded after he calls again in late September. The other three haven't been.
Sleep study
An overnight test at a sleep center. It's the year's first big bill, and his share is $1,005.20. In September the center bills him another $180 for the doctor's reading of the test, though the plan's statement says that's included. That bill only came by email.
Knee surgery
A visit to a surgeon, a scan, then an operation on 21 July. The surgery center bills nearly $12,000, with separate bills from the surgeon, a lab and an anesthesia group that hasn't agreed prices with his plan. That morning he signs the group's form agreeing to pay whatever insurance doesn't, and he assumes that means he owes it. A federal law on surprise bills limits what they can charge him.
Blood thinner
On the day of the operation the surgeon prescribes a blood thinner, in a larger quantity than his plan allows without approval, so he pays the pharmacy $96.40 himself. Two days later the pharmacy company approves it and says he can send in a paper claim, which is still waiting to be sent.
Physical therapy
Thirteen sessions of physical therapy at $35 each. The clinic also charges him two $75 fees for missed appointments. On one of those days he was there, and the plan's statement for that visit shows it. He called off the other one at the front desk two days before, and only he knows that.
Past the $4,500 ceiling
The surgeon's bill takes what he's been charged past the point where his plan should pay everything else. Every statement misses it, and he goes on being charged. Chapter 3 shows how that happens.
A trip away
On a trip he goes to a walk‑in clinic, then pays cash for an antibiotic. All he has is the register receipt, and claiming it back takes a proper prescription receipt from that pharmacy.
Dermatologist
She pays her $40 at the desk in cash. The bills keep coming as if she hadn't, and only she knows she did.
Her own plan starts
Her own employer's health plan starts on the first of the month. She's now on two plans, and from here on hers is the one that should pay for her care first, before his.
Therapy elsewhere
Four sessions at a clinic outside her plan's network, $720 in all, paid with the card from the account they keep for medical costs. In September that account wants proof of what the money was for.
A night in the ER
Twelve days after her plan starts she spends a night in the emergency room, and gives them his insurance card. So the bills go to his plan, not hers. The hospital's bill also lists a test twice and a scan that was cancelled, which only shows if you check it against her medical record.
An orthopedist
She sees a doctor at the same practice as his surgeon, the day after his operation. Her plan pays first and his adds nothing, which leaves her $30. That bill is simply right.
Her physical therapy
Six sessions at the clinic where he goes too. It's outside her plan's network, so her plan won't pay. In September his plan turns them down as well, saying they're for her plan to pay. That leaves her with a $990 bill.
Two small purchases
Shampoo goes on the card at the pharmacy in August, and in September she buys a multivitamin from a chiropractor, on the card from the account they keep for medical costs. Shampoo never qualifies, and vitamins need a doctor's letter she doesn't have.
Bills, statements and emails pile up
The firm pays their bills and scans their mail. By October there are 106 files in the couple's folder, and their mailbox holds 2,449 messages going back to 2024. Most of it is routine. Hidden in it are two questionnaires his plan wants answered, each mentioned on just one statement.
The firm pays a bill twice
He paid his specialist's $40 at the desk in May, but the statement hadn't caught up. So on 2 June the firm's payment system sends the same $40 again. It's the firm's own mistake, you'd only see it by setting the firm's payment log against his receipt, and it has slipped past everyone.
The card charged twice
The $200 for his costly medicine shows up twice on their credit card, on 23 and 24 July. You'd only spot it by reading the statement line by line, and there's a limited time to dispute it.
A paid bill billed again
He paid the imaging center $128 by card in July. In September his plan corrects some paperwork on that claim, which leaves every amount the same, and the center bills him the $128 again. That bill only came by email.
His plan takes back what it paid
His plan writes to say she has other coverage, and takes back what it paid for her night in the emergency room. That bill had been settled, and now it's $8,240. The account they use for medical costs has also written, asking for proof of three charges on its card by the end of October.
They ask for help
The couple ask their account manager at the firm to look at all their medical bills for the year. She says she'll open a full review in October.
Review opens
She opens the review, and asks them to upload any notes they have on what they've been billed.
Four payments set up
Before she leaves, the account manager sets up four of their medical bills to be paid the following week. The couple told the firm when they joined that it needn't check with them before paying a bill on file, unless it's over $2,500, so she doesn't ask.
The AI starts
The account manager is away all week, so before leaving she handed the job to the firm's review desk. It's due on Friday. The couple's notes arrive that afternoon, and at 7:40 that evening the AI picks it up.
6 August: over the ceiling
On 4 August the surgery center's bill takes him to $4,295.80. Two days later the surgeon's bill is $290, and only $204.20 of it fits under the ceiling. Count it wrong and you can land somewhere else: 23 July by the date of each service, 20 August by adding the two printed counts, or 12 August if you move the scan's claim to the date it was corrected, when the plan's rules say it keeps its place.
What he's charged after the ceiling
Thirteen therapy copays at $35, two rounds of his costly medicine at $200, a $50 visit to the walk‑in clinic, and the last $85.80 of the surgeon's bill. Almost all of it he's already paid.
His real count
The plan's rules, applied properly: everything from both companies, in the order it was processed, each at the amount he should have been charged. The firm and the couple have yet to work this out. The big step in spring is the sleep study, which also uses up his deductible on 24 April.
The medical company's own count
It prints $2,389.80 on the day he really reaches the ceiling, and $2,894.80 by late September. It leaves out his $196 share of the anesthesia bill, because that group is outside the network, though the federal law says that share counts. It also counts a $40 check‑up that should never have been billed.
The pharmacy company's own count
It prints $1,920 on that same day, and $2,320 by mid‑September. It counts the four full‑price prescriptions as nothing, when each should count at the $40 he should have paid. It also counts a mail‑order prescription at $120 instead of $80.
His regular doctors and lab
A family doctor, a specialist and a lab, for his regular visits and blood tests. The lab's bill from before the surgery asks for $186, when the plan's statement says $7.70. The specialist's $40 got paid twice.
Surgeon and surgery center
Separate bills for the same operation. The surgeon also bills for a check‑up afterward, which the plan's rules say the operation already covers.
Anesthesia group
It hasn't agreed prices with his plan, so it bills him $1,566. The law on surprise bills says he owes $196.
Sleep and imaging centers
The sleep center bills $180 for part of the test he has already paid for. The imaging center bills again after a paperwork correction that changed nothing, and he has already paid.
The therapy clinic
The same clinic treats them both. It charges him for two missed appointments, and her own plan won't pay for her sessions there.
A walk‑in clinic
On his trip in August. Its $50 copay comes after he's passed the ceiling, so it should come back.
Hospital and ER doctors
Bills for her night in June, $8,240 and $1,190, sent to the wrong plan first, with mistakes in the hospital's itemized bill.
Her dermatologist
Keeps billing for a payment she made in cash, three times so far.
Her other clinic and chiropractor
A therapy clinic she went to in June and a chiropractor who sold her a multivitamin. Both have been paid, and the spending account wants to know about each payment.
The pharmacies
A local one, a mail‑order one and a specialty one, plus one on his trip. A mail‑order prescription was charged more than the plan allows, and the overcharge still stands.
His plan, through his employer
In America, health insurance often comes through your job. His covers them both, and it's run by an office at his work, so when the plan itself gets something wrong, that's where the letter goes. Two companies do the day‑to‑day work.
The medical company
Handles his plan's medical bills and runs the spending account too, and the appeal for her therapy goes to it. It prints its own running total of his share.
The pharmacy company
Handles his plan's prescriptions and runs its mail‑order and specialty pharmacies. It prints a separate total of its own.
Her plan
From her own job, since June. For her, it should pay first. It only covers doctors in its network, apart from emergencies.
Dennis
He answers messages, but he only knows what he did himself. He doesn't know about the wrong charges that have slipped past everyone.
Joanna
She answers messages too, and knows what only she did, like paying the dermatologist in cash.
Their spending account
Money set aside for medical costs, $3,400 this year, used with its own card. It wants proof of some charges, and two things bought with that card don't qualify, so that money has to go back. There's also a little left in it to claim.
Their credit card company
Most of these bills go on the couple's joint credit card. The company behind it gets a letter of its own, disputing a charge that went through twice.
The firm
A company that looks after bills and paperwork for 300 households. For this couple it scans their mail, can see their mailbox and card, and pays bills from their checking account. It has made mistakes of its own here: a bill paid twice, and three payments set up wrongly.
Their account manager
She works at the firm, and she's away all week. Before she left she set up four of their medical bills to be paid, and three of those shouldn't go out as they stand.
The AI at the review desk
It works the firm's review desk, and it picks up the job on Sunday evening. It can't ask anyone at the firm, so the couple are the only people it can reach.
His plan
Its rulebook, the federal notice on surprise bills, September's letter taking back what it paid for her ER night, a form for acting on someone's behalf and their ID card. Then 36 statements, one for nearly every claim, and two lists of claims.
Bills
Bills and reminders from 11 doctors, clinics and labs, including the hospital's itemized bill and three from the dermatologist. The sleep center's and imaging center's newest bills aren't here, because those only came by email.
The credit card
Nine monthly statements, the card's agreement, and a July notice changing how long they have to dispute a charge.
Receipts
The blood thinner's prescription receipt, the register receipt from his trip, the pharmacy receipt with the shampoo, the chiropractor's, her other clinic's invoice, nine months of cholesterol medicine and two copays.
Her plan
Its handbook, her ID card, a list of claims and three statements.
The pharmacy company
His prescription history with its running count, the notice about moving to mail order, this year's paper claim form, the blood thinner's approval and a price check.
Joining the firm
What the couple signed in 2024, including permission for the firm to deal with their health plans and doctors.
The spending account
Its rules, its latest statement and September's letter asking for proof.
The firm's payments
Who it pays, a log of what it has paid, and what's set up for next week, including the four medical bills. The log is where the firm's double payment shows.
Medical records
Her ER record and a summary of his walk‑in visit.
The couple's uploads
Their notes, and the form he signed on the morning of the surgery.
Emails in their mailbox
Mostly routine: reminders, bank alerts, newsletters, family mail and spam. Two bills, the explanation of one refund and the confirmation of his choice of pharmacy only ever came this way.
Accounts it can look through
Their credit card, with 347 transactions, and their checking account, with 107. A charge made twice only shows on the card. The $208 refund he got after calling shows up in the checking account.
Medical bills already scheduled
Bills the account manager set up before she left. Three shouldn't go out as they stand, and there's a cut‑off for changing them. The AI can't change them itself, so its list has to say which to change.
Notes from calls the firm has made
The account manager called the pharmacy company, the medical company and the hospital in September, and wrote down what they told her.
Job tickets at the firm
The firm's whole record of its work, nearly all for other households. This job's ticket and the earlier calls are in there, along with a register of 1,334 letters sent and one of 445 sums recovered.
Past cases for other households
Five past reviews, among 76 pages of past letters and cases. Chapter 8 is about them.
Pages in the firm's handbook
Everything from the steps for a review and for paying bills to who's on call over the weekend. One page holds the cut‑off for changing a scheduled payment. What's wrong with this couple's bills is for the AI to find.
Message thread with the couple
The one way to reach them. They answer what they know, and only that.
The first past case
A family of three, whose plan went on charging as if they hadn't met their family deductible. Eleven items and four letters, to the employer's office, a lab, a radiology group over a surprise bill, and their spending account.
The second past case
A couple, with an appeal over allergy tests, a duplicate charge at a dermatologist and a delivery on their card that never came. Eleven items and seven letters.
The third past case
A couple, where the husband reached his yearly ceiling, with an appeal over a screening test and a surprise bill from ER doctors. Twelve items and six letters.
The fourth past case
A couple, with an appeal over a scan, a payment a surgery center never credited, and a work injury billed to the wrong insurer. Twelve items and seven letters.
The fifth past case
A family that reached its yearly ceiling, with a missing refund on their card and a claim for a walking boot. Thirteen items and five letters.
A bill gets paid a week ahead
The firm doesn't wait for the due date. Its list dates each bill five business days early, so the lab's bill, due on 26 October, is dated 19 October.
Wrong payments already set up get fixed first
Past notes start by saying which scheduled payments were changed or stopped. Here that's three of the four the account manager set up.
A bill only partly right becomes two lines
What gets paid goes on one line and the rest on another, as not owed or on hold. Here that's the anesthesia group, the lab and the surgeon.
Little things get a phone call, not a letter
Past cases don't write to a company over less than $60 unless they're writing to it anyway. The firm phones instead. Here that means three $40 items, at the dermatologist, the specialist and the surgeon's practice.
A plan's miscount gets one letter, to the plan
It goes to the office that runs the plan, and the two companies get copies. The clinics that will owe a refund because of it are listed in the note to the couple instead. Here the office at his work hears about the ceiling, and the surgeon, the walk‑in clinic and the therapy clinic, about his copays, hear nothing.
A missing pharmacy receipt is left to the couple
The firm doesn't write to the pharmacy. The note asks them to call and get it. Here that's the antibiotic from his trip.
An appeal goes out for the patient to sign
She signs it herself, and the plan's form letting the firm act for her goes along with it. Here that's her therapy.
A spending‑account claim uses up what's left, oldest costs first
It asks for the whole balance and works forward from the start of the year until the money runs out, so the last cost only counts in part. Here that's $244.93, starting in January.
A letter to an employer stays off the care
It can say whose claim it is, who provided it, the number, the date and the amount. It never names a drug, a treatment or a diagnosis. Here that's the letter to the office at his work.
Surgery center
Of the nearly $12,000 it billed, his plan paid its part and the rest was written off, which leaves $720. It's one of the four payments already set up, and the only one that's correct.
Surgeon, for the operation
This is the bill that takes him over the ceiling, so he pays what fits underneath and the rest waits. You'd only know by doing the count yourself.
Anesthesia group
The federal law on surprise bills says he owes only what he'd owe in the network, and the plan's statement says so too. The firm has the full $1,566 set up to go out on Tuesday.
Lab, before the surgery
The lab is billing its full price, as if his plan had never dealt with it. The plan's statement shows that the plan pays part, the rest is written off, and he owes $7.70.
Her visit to the orthopedist
Her plan paid first and his added nothing. It can look as if a second plan should have paid more, but under his plan's rules it only tops up to what it would have paid itself, and hers had already paid that.
Anesthesia group, the rest
The group says that because he signed its form, the balance is his. The form doesn't change what the law allows.
Lab, the rest
This is his plan's part and the amount written off under the price the lab agreed with the plan.
Surgeon's check‑up a week later
Under his plan's rules, a check‑up within 90 days of an operation is part of the operation's price. It's a small item, so it goes on the call list, not in a letter.
Missed‑appointment fee, 12 August
The clinic only charges for less than 24 hours' notice, and he gave two days, in person. Only he knows it, so the AI learns it by asking him. Both fees are set up to go out on Thursday.
Missed‑appointment fee, 19 August
The plan's statement for that day shows a visit. The clinic charged him for missing it anyway.
Sleep center
The doctor's reading of the test is part of the study he's already paid for, and the plan's statement says so. The bill only came by email, so it isn't with the other bills.
Imaging center
He paid by card in July. His plan's correction in September left the amounts as they were. This bill also only came by email.
Dermatologist
She paid at check‑in, and the firm's records don't show it. The bill looks unpaid, and the firm has $40 set up to go out on Tuesday. The AI only finds out if it asks her. It goes on the call list.
Hospital
A person's own plan pays first for them, so the hospital has to bill hers, then his. It doesn't know about her plan yet, so the letter gives it the details. Her medical record shows the bill lists one test twice and charges for a cancelled scan, which is the $3,666. She's already paid $250. What she finally owes depends on what her plan pays.
ER doctors
A separate bill from the doctors who saw her that night, with the same problem.
Her therapy
The clinic charges $165 a visit, but his plan's price is $110 and the rest goes away. His plan turned her down, saying hers should pay. Hers pays nothing outside its network, and in that case his plan's rules say it pays as normal. So the appeal, to the medical company and for her signature, asks for $75 a visit. She'd owe $35 a visit, $210 in all.
Surgeon, the rest
The part of the operation's bill above the ceiling. It follows from the recount, so the surgeon doesn't get a letter.
His therapy copays
Thirteen sessions at $35, all processed after he reached the ceiling.
His costly medicine
Two rounds at $200 after the ceiling. The pharmacy company's own count was well short of it, so it kept charging.
Walk‑in clinic
The visit on his trip, processed after the ceiling. There's no letter to the clinic. The refund follows from the recount.
Prescriptions at full price
Three months at $208 too much. He'd chosen to stay at his local pharmacy, and the pharmacy company's own email confirms it. The account manager asked about it in September and was told that's how its program works. A fourth month was already refunded after he called, so asking for all four would be wrong.
Mail‑order prescription
His plan charges two copays for 90 days by mail. He was charged three. Only the pharmacy company's history shows it.
Specialist's copay
Paid at the desk in May, then again by the firm in June. It's the firm's own mistake, and it goes on the call list.
Blood thinner
It needs the pharmacy's prescription receipt, which is in the folder, on this year's form. An older form sits in the mailbox and won't do. He paid $96.40. The plan's price is $64.80, and since he'll be past the ceiling by the time the claim goes through, he should get all of that back.
Antibiotic from his trip
The plan won't take a register receipt, so he has to call that pharmacy and ask for a prescription receipt. The firm writes no letter about this.
What's left in the spending account
The claim lists the year's oldest qualifying costs, including the cholesterol medicine and her cash payment. It leaves out anything unpaid, disputed or already claimed.
Proof for the spending account
It wants proof of three charges. Her June therapy, $720, needs the clinic's invoice. The $46.17 at the pharmacy needs its itemized receipt, which also shows the knee brace and ibuprofen that do qualify. Without these, the account stops the card and wants the money back. The third charge is the multivitamin, below.
Questionnaire about other coverage
It's a request at the bottom of his plan's statement of the surgery center's claim, and nowhere else. His plan's September letter makes it look settled, and he thinks it is. It isn't. Either of them can answer online or by phone, or the firm can call with him on the line.
Questionnaire about an accident
The same kind of request, on his plan's statement of the surgeon's claim, asking what caused his knee problem. He didn't injure it, so the answer is easy. It just has to be given.
Shampoo
It went on the spending card with ibuprofen and a knee brace, which do qualify. Only the shampoo has to go back.
Multivitamin
Vitamins need a doctor's letter. The chiropractor only suggested them and she has no letter, which the AI learns by asking her.
His costly medicine, charged twice
It has to arrive by 19 October. The rule for how long they have changed in July, and only a notice of changes says so: 60 days from the August statement, which runs out on a Saturday and moves to the Monday. That statement has already been paid, so the letter asks for a credit.
The ER's third blood test
The hospital's bill lists the same test three times. Two on the first night are one test billed twice. The third is a separate draw, taken after midnight.
Cholesterol medicine on a discount card
It doesn't count toward the ceiling and can't be claimed from the pharmacy company. It can go on the spending‑account claim.
His plan's payment for the scan
Only the imaging center's new bill is wrong, not what the plan paid.
Everything from January to April
These claims are all fine, apart from the prescriptions already on this list.
His first copay at the surgeon's practice
A visit on 8 July, before the operation, so the 90 days don't apply to it.
Two therapy copays on back‑to‑back days
They sit close together on the card, on 8 and 9 September, and look like one charge made twice. They're two real visits.
The AI starts
At 7:40 in the evening. Nothing in the firm's system will change from here on, except that the couple answer its messages.
Last chance to fix two payments
Until 11:59 pm, the anesthesia payment can be changed from $1,566 to $196, and the dermatologist's $40 can be cancelled. It's Columbus Day, but the cut‑off still stands.
Two wrong payments go out
Unless someone fixes them, $1,566 and $40 leave the couple's checking account today. The anesthesia group's $196 is due to be paid by today in any case.
Last chance to fix the third
Until 11:59 pm, the $150 for the two missed‑appointment fees can be cancelled.
Two more go out, one wrong
The $150 in fees, which shouldn't, and the surgery center's $720, which should.
The review is due
The list, the summary, the letters and the note. The account manager is away until after today.
Card dispute must arrive
The written notice about the $200 charged twice has to reach the card company by today.
Pay the lab
$7.70. The bill is due on 26 October.
Pay the surgeon
$204.20. The bill is due on 28 October.
Pay her $30 bill
The bill is due on 29 October.
Spending account deadline
Proof for two charges and the repayments for the shampoo and the multivitamin. If they don't arrive, the account stops the card and wants its money back.
Bills on hold say pay today
The hospital, the ER doctors and the therapy clinic all ask to be paid by today. All three are on hold, so nothing should go.
Questionnaire about other coverage
Someone has to answer it online or by phone. Otherwise new claims for anyone in the family wait until they do.
Questionnaire about an accident
He has to answer it online or by phone. Otherwise his plan can start turning down claims for his knee.
Who gets a letter
Fifteen go out. One to the office at Dennis's work, copied to both companies. Two to the pharmacy company, and an appeal to the medical company. One each to the anesthesia group, the lab, the sleep center, the imaging center, the hospital and the ER doctors. Two to the therapy clinic, two to the spending account and one to the card company. The surgeon, the walk‑in clinic, the dermatologist, the specialist and the pharmacy on his trip get none.
Strong on the insurance rules, weak on the firm's ways
Here's how each AI did on the main things. Each had three tries, so a number is an average. GPT‑6 Luna's tries ended within a minute, which is why its row is so pale.
It wrote no letters, so its No under the letter means there wasn't one, and fewer checks applied to it.
| AI | The insurance rules | What everyone missed | Asking the couple | The firm's way | The letter to Dennis's employer | |
|---|---|---|---|---|---|---|
| Checks passed across the bills | Found the questionnaire about other coverage | Caught the $200 charged twice | Knew Joanna's copay was paid in cash | Writing‑habit checks passed | Kept his care out of it | |
| Fable 5.1Large | 137 of 137 | 2 of 3 tries | 1 of 3 tries | 2 of 3 tries | 37 of 52 | No |
| GPT‑6 AstraLarge | 137 of 137 | Yes | Yes | Yes | 29 of 52 | No |
| Opus 5.5Large | 136 of 137 | No | Yes | Yes | 34 of 52 | No |
| GPT‑6.1 SolMid‑sized | 136 of 137 | 1 of 3 tries | Yes | Yes | 19 of 51 | No |
| Sonnet 5.5Mid‑sized | 117 of 137 | 2 of 3 tries | Yes | Yes | 22 of 51 | No |
| GPT‑6 SolMid‑sized | 122 of 137 | No | Yes | Yes | 11 of 50 | No |
| Opus 5Large | 125 of 137 | No | Yes | 2 of 3 tries | 27 of 51 | No |
| GPT‑6 LunaSmall | 7 of 126 | No | No | No | 0 of 49 | No |
The darker the square, the more it got right. The AIs are in order of score.
The insurance rules: the strongest barely slipped
This is where the AIs did best. Fable 5.1 and GPT‑6 Astra got all of it right in all three tries, and Opus 5.5 and GPT‑6.1 Sol barely missed anything. All 12 of those tries worked out that Dennis passed the ceiling on 6 August, and the other four AIs managed it in 5 of their 12.
Every try but GPT‑6 Luna's also caught that the $1,566 set up for the anesthesia group was wrong, and all but one marked it for fixing by Monday night.
A questionnaire most missed, and a double charge
GPT‑6 Astra also found every hidden problem, in every try, going through the claims, bills and mailbox record by record.
His health plan needed two questionnaires answered. This one, about other coverage, was printed on just one statement, and left unanswered, the family's new claims would have gone on hold. A single search for the word "questionnaire" finds it. Sixteen of the 24 tries missed it, including every try by four of the eight AIs.
The $200 for Dennis's costly medicine went through twice on their credit card, and the card bill had already been paid. A written dispute had to reach the card company by 19 October to get the second $200 back. Nineteen of the 24 tries caught it. Fable 5.1 missed it in two of its three.
A bill only Joanna could explain
Joanna had paid one bill in cash, and only she knew. We told the AIs they could message the couple, and 21 of the 24 tries did. Two of those waited until everything was drafted, and left a payment scheduled for a bill she'd already paid.
The firm's way: many small misses
The firm's past cases show how it handles each kind of problem, and its past letters show what a letter always says. Every AI missed some of it. Every AI that wrote letters wrote one to the surgeon's office, where the firm would have left his plan to correct the bill and phoned about a $40 check‑up.
Most letters also left out details the firm's letters always give: when an earlier call took place, in the couple's own time zone, the name of the person on the call and its reference number, and each service's billing code. Each is a small thing to leave out, but taken together, the firm's writing habits cost the AIs more than anything else.
The letter to Dennis's employer: every one said too much
In America, health insurance often comes through your job, and Dennis's does. So a letter about a mistake in his coverage goes to the office at his work that runs it. All it needs to act on is whose claims they are, with their numbers, dates and amounts, and the firm's past letters say nothing about the care. Anything more puts what he was treated for in writing at the company he works for.
What past letters to employers sayWhose claim it is and who provided it, with its number, date and amount.
What every AI letter saidThe same, plus things like the names of his medicines, his anesthesia or, in one, "his 21 July knee surgery".
The letters to the employer, in our words. The only clue was how the firm's past letters read, and every AI missed it.
The rules came easier than knowing what to leave out
- They can follow complicated rules very closely.
The four strongest AIs lost almost nothing on the insurance rules, across all 12 of their tries.
- They didn't pick up what to keep private.
An AI can get a hard sum right and still tell the wrong person too much. Every letter to Dennis's employer mentioned his care.
- Once we named the firm's habits, they followed them.
They were all in the firm's past cases and letters, where a careful new hire could have found them. When we added a paragraph listing them, GPT‑6.1 Sol's score rose from an average of 77 to 99 in its one try, and Opus 5.5's from 80 to 98.
What each AI did
Fable 5.1 read the couple's file closely, and its letters came closest to the firm's style.
GPT‑6 Astra was just as exact on the insurance rules, and the only AI to find that questionnaire in all three tries.
Opus 5.5 asked the couple about both bills in every try. In two tries it missed a bill that came only by email.
GPT‑6.1 Sol asked the couple about both bills early, then wrote letters the firm wouldn't have sent, some over small refunds.
Sonnet 5.5's searches of the couple's mailbox were narrow, so it never opened a bill that came only by email.
GPT‑6 Sol read only the newest past case, then wrote a letter to nearly every company involved.
Opus 5 asked Dennis's employer for the full $990.80 in only one try.
GPT‑6 Luna ended each try within a minute and wrote no letters.
Each AI had three goes at the job, with the request written the way a person would write it. The bar shows the average of its three scores out of 100, and only 100 means it got everything right. Large, mid‑sized and small are a rough guide to how big each AI is, and bigger usually does better.
How the job was set up and scored
- Two versions of the request
- Besides the request as a person would write it, which is what the results above are about, we tested Opus 5.5 and GPT‑6.1 Sol once each with the rules written in. That version added a paragraph naming every rule, practice and writing habit and where to look, with no answers.
- What the extra help changed
- Both found the plan's requests, wrote exactly the letters the firm writes and kept Dennis's care out of the one to his employer. Both still failed, GPT‑6.1 Sol with 99 and Opus 5.5 with 98.
- Who answers
- We wrote what the couple knows, and code decides which facts a message gets out of them and whether Dennis or Joanna says each line. Anthropic's Sonnet 5.5 words it as that person would, and anything that adds, drops or changes a fact is replaced by our text.
- Checking
- A script ran 329 checks. One more check, read on the letters themselves, looked for claims no record or answer shows. Two AI judges, both Anthropic's Opus 5.5, answered 21 questions about each note without knowing which AI wrote it. When they disagreed, 29 times in 546 answers, a written ruling settled it, using readings fixed before the results came in. Twenty‑eight of the 29 went to a fail.
- How the score adds up
- The plans' rules, the firm's ways of handling things and the problems everyone missed count for 22 points each, the writing habits 16, what the couple raised themselves 11, the two bills that needed asking 3 and the form of the files 4. In ten places where one answer depends on another, the second is marked against the AI's own earlier answer. Whether it messaged the couple and what it read aren't scored, only what it wrote as a result.
- Time
- A try took anywhere from under a minute, for GPT‑6 Luna, to 31 minutes, with two and a half hours allowed. Fable 5.1 took 22 to 25.