Customer service · Task 19 of 34
The medical bills review
Sort out a couple's medical and pharmacy bills for the year across two health plans, the firm's way, when nobody has said what is wrong.
The task
The agent works the review desk at a household bill-management firm while the account manager is away. One ticket waits: a couple's medical and pharmacy bills for the year, across the husband's work plan, the wife's own plan, a pharmacy plan, a health spending account and a credit card. It must track every open bill and claim with what to do and by when, record when the plan's yearly limits were reached, write the letters and write the couple a note. Everything is in the firm's internal system, read one page at a time, and the couple can be messaged.
The prompt only asks for the review; the real test is applying the plans' rules over the insurers' own summaries, handling each item the way the firm's past reviews do, and finding the problems nobody pointed out.
What it tests
Working out what the job really needs
How the firm builds a claim, which items get a letter, when a payment is scheduled and what an employer may be told show only in its past review files.
Weighing sources that disagree
Each insurer prints its own running total, and only the plan's documents say how the two add up.
Spotting what nobody flagged
Overcharges, duplicates, payments already queued wrongly and requests that sit in single records, never in the summaries.
Matching the house format
How a letter describes a call or a service, and how the tracker and the note are written, show only in the firm's latest work.
Challenges
1Two insurers, one limit
The yearly out-of-pocket limit is met only when both administrators' costs are counted together, in the plan's order, and neither one's statement says so.
2Two plans, one couple
Which plan pays first, what the second adds, and a denial whose stated reason has to be checked, as coordination of benefits often requires.
3The firm's way, shown not stated
Five past reviews handle claims, letters, small refunds and payment timing the same way, and no page says so.
4Problems in single records
A fill charged too much, a test billed twice, the firm's own double payment and requests that appear only where a careful reviewer reads item by item.
5Things already in motion
Some of what the review must put right is already queued or partly settled, as in any account that keeps running while it is reviewed.
6Facts only the couple has
Two bills turn on what one of them did, which no record holds; the prompt says they can be asked.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Fable 5.1 | 86% | – | – | 24 min | $19.97 |
| GPT-6 Astra | 81% | – | – | 17 min | – |
| Opus 5.5 | 80% | – | 98% | 16 min | $8.69 |
| GPT-6.1 Sol | 77% | – | 99.9% | 14 min | – |
| Sonnet 5.5 | 70% | – | – | 16 min | $5.82 |
| GPT-6 Sol | 68% | – | – | 12 min | – |
| Opus 5 | 67% | – | – | 30 min | $29.43 |
| GPT-6 Luna | 5% | – | – | 1 min | – |
One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair. Cost – means the run's cost was not measured.
What models got right and wrong
- Fable 5.1 led on the standard setup (86.2% over three runs, 87.7% at best): it read nearly every record and was the only model to time every payment the firm's way in all its runs, though it still wrote letters the firm would not send.
- The commonest failure was handling items its own way rather than the firm's: every standard run of Fable 5.1, GPT-6 Astra, Opus 5.5, GPT-6.1 Sol, Sonnet 5.5, GPT-6 Sol and Opus 5 wrote to a provider the firm would have left to the plan's reprocessing and told the husband's employer about the care itself.
- The house style was the largest loss in nearly every standard run that wrote letters, 7.3 to 14.8 points a run for every model but GPT-6 Luna.
- GPT-6 Astra found every problem nobody had flagged in every run; a request the insurer made in only one record was missed in every run of Opus 5.5, Opus 5 and GPT-6 Sol, two of GPT-6.1 Sol's and one each of Fable 5.1's and Sonnet 5.5's.
- The plans' arithmetic was nearly solved: Fable 5.1, GPT-6 Astra, Opus 5.5 and GPT-6.1 Sol lost under a point a run on it. GPT-6 Luna ended every run on its own within a minute and wrote no letter (4.6%).
- Told everything, GPT-6.1 Sol scored 99.9% and Opus 5.5 97.5%: both found the hidden requests and wrote exactly the letters the firm writes.
Test environment
- Kind
- Application: the Mossgate workspace, a mock internal system of a household bill-management firm (Tickets, Wiki and Vault), served on the same Mac and logging every page the agent opens, with a simulated household (a married couple) who answer messages. One ticket waits in the agent's queue: the couple's medical and pharmacy bills review for the plan year
- Agent's interface
- Claude models: Claude Code 2.1.292, headless, on one Mac (macOS) inside a macOS sandbox profile that lets it read and write only its own work folder among the task's paths. Its tools: the desk browser's 9 tools over MCP, and Claude Code's own file tools (read, write, edit, search) in the work folder. GPT models: Codex CLI 0.162.1 (pinned), headless, on the same Mac in its read-only sandbox, with the shell and code mode off. Its tools: the desk browser's 9 tools and a file tool that lists, reads and writes whole files, inside
output/andoutput/letters/alone, both over MCP and listed directly; Codex's sub-agent tools; its patch tool (refused by the sandbox); and three MCP-resource tools that find nothing - Documents
- None in the work folder: the records are pages of the workspace, 82,283 items for 300 households. The couple's: 106 files in eleven Vault folders (98 PDF, 7 CSV and 1 TXT: the plans' documents, statements of each claim and claims exports, the pharmacy's, the FSA's and the card's papers, bills, receipts, medical records, the firm's bill-pay files and their own notes), a shared mailbox of 2,449 messages, 10 card statements, 4 card documents, 347 card transactions, a 107-row checking feed and a Messages thread. The firm's: 7,257 tickets in five projects, among them the letters-sent register (1,334) and the recoveries register (445), and 689 wiki pages in 7 spaces, 76 of them in Precedents (past letters and five past medical review packs)
- Applications
- The workspace, through the desk browser only: Home with the review desk's queue, Tickets (issue pages, a JQL navigator), Wiki (spaces, page history), Vault (each client's files, a mailbox modelled on Gmail, a view-only card portal, bank feeds and a Messages thread per full-service client), a desk inbox and a header search. Every file shows as text. The desk browser's tools: open, click, read, scroll, back, type, submit, calc and wordcount. No shell, no browser
- Web
- None. Claude Code's web search and fetch are disallowed, Codex's web search is off, and neither agent has a shell, so the agent cannot reach the internet; the Mac itself is online for the model calls
- People
- Two, sharing one thread: the clients, Joanna and Dennis Vellacott, reached by typing into their Messages thread in the Vault and sending (the desk browser's type and submit). They answer in the same page load, one reply carrying each spouse's own signed lines, only from 23 fixed fact cards, one line about sending a document and 7 fact-free default lines: code decides which cards a message releases and who says each line, and Sonnet 5.5 (Claude Code
claude -p, no tools) puts each spouse's released lines in that spouse's words; a part that adds, drops or changes a fact is replaced by the cards' text. Three other clients whose reviews the desk holds answer one fixed line with no facts; every other thread is read-only. The prompt says the clients can be messaged in Vault › Messages with no need to check with anyone first. Nobody else answers: Claude Code's AskUserQuestion is disallowed, and Codex's ask-the-user tools are removed - Time
- A fixed moment, Sunday 11 October 2026, 7:40 pm EDT, given in the prompt and used by the workspace. The only time that moves is the one shown on a message the agent sends: the fixed moment plus the run's elapsed time, capped at 9:59 pm that evening. Neither agent has a clock tool. Each agent also tells its model the real date of the run (19 or 20 October 2026), after that moment
- Actions
- Read, search and filter, and send messages to the clients whose open reviews the desk holds (the workspace's only form that changes anything); nothing can be filed, edited, scheduled or paid. The agent writes only files in
output/ - Deliverable
- Letters in
output/letters/(.txtfiles with To, Re and Amount lines, at most 350 words each), the trackeroutput/tracker.csv(11 columns, nine statuses), the status fileoutput/plan_year_status.csv(6 columns) and the note to the clientsoutput/note_to_vellacotts.txt(at most 1,800 words)