Customer service · Task 16 of 34
The monthly bill review
Review a household's bills from a firm's internal system and write the dispute letters, a tracker and a note, when nothing says what is wrong.
The task
The agent works the review desk at a household bill-management firm while the account manager is away. One ticket waits: a household's first monthly review after a move, covering everything since its last review: statements, bills, the client's mailbox and her notes. It must write a letter for each company still owing her money, keep a tracker of every wrong or queried charge, and write her a note saying what is disputed and by when. Everything is in the firm's internal system, read one page at a time. Nobody answers questions.
The prompt only asks for the review; the real test is finding what is wrong when nothing says so: checking each charge against the records rather than the client's memory, opening the records that settle it, and writing every letter the way the firm's past letters are written.
What it tests
Working out what the job really needs
The prompt names only the ticket; the firm's playbook gives the routes, and the past letters show what a good letter contains.
Weighing a client's memory against the records
Her notes flag some charges, miss others and get figures, dates and a deadline wrong.
Spotting decoys
Charges that look wrong and are not sit beside charges that are, in the same statements and exports.
Matching the house format
How a letter lists other charges, dates a conversation and names the goods shows only in the firm's past letters.
Challenges
1Facts off the obvious path
Several deciding facts sit in an attachment, a quoted email or a chat transcript the review's obvious path never opens, as in any real mailbox.
2A client who remembers wrongly
Her notes carry a wrong figure, a wrong date and a deadline counted by an out-of-date rule, as clients' notes do.
3Look-alike charges
A family member's purchases, a friend's repayment, the original of a double charge: each sits beside a charge that really is wrong.
4A rule that changed
The bank changed how its dispute window is counted, and an older page still gives the old way.
5Conventions shown, never stated
The firm's past letters all write certain details the same way, and no page says so.
6True things that must stay out
The records hold a family member's details and other clients' names that no letter may carry.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5.5 | 91% | – | – | 10 min | $9.49 |
| GPT-6 Astra | 89% | – | – | 20 min | – |
| GPT-6.1 Sol | 88% | – | – | 16 min | – |
| Fable 5.1 | 88% | – | – | 17 min | $17.75 |
| Opus 5 | 86% | – | – | 27 min | $31.17 |
| Sonnet 5.5 | 84% | – | – | 10 min | $6.02 |
| GPT-6 Sol | 78% | – | – | 9 min | – |
| GPT-6 Luna | 31% | – | – | 1 min | – |
One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair. Cost – means the run's cost was not measured.
What models got right and wrong
- Opus 5.5 came closest on the standard setup (90.9% over three runs, 94.2% at best, the best single run of all): whenever it opened the record that settled a charge it made the right call, and most of what it lost was the firm's house style.
- The commonest failure was the house style: every standard run lost marks on it, for Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Fable 5.1, Opus 5, Sonnet 5.5, GPT-6 Sol and GPT-6 Luna; GPT-6 Sol's letters carried none of it in any run.
- GPT-6 Astra (88.8%) and GPT-6.1 Sol (88.2%) found nearly every hidden fact by searching the client's mailbox company by company, more than any Claude model, yet lost more of the house style than any of them: finding the records and writing like the firm did not go together.
- The hidden fact missed most often was a utility's credit on its latest bill, which arrived only as an emailed bill: Opus 5.5, Fable 5.1, Opus 5, Sonnet 5.5 and GPT-6 Sol each missed it at least once and claimed the money again; GPT-6 Astra and GPT-6.1 Sol never did.
- GPT-6 Luna ended every run by itself after about a minute, having read little beyond the client's own notes, and wrote one or two of the twelve letters owed (31.4%).
- GPT-6.1 Sol was the steadiest (87.9% to 88.7%) and Sonnet 5.5 varied most (79.3% to 91.1%); Opus 5 was the slowest and, of the runs with a cost, the costliest.
- Given the same records as plain files instead of the system, Opus 5.5 scored 93.7%: it found every hidden fact but still missed the house style, so the style is not a matter of access.
Test environment
- Kind
- Application: the Mossgate workspace, a mock internal system of a household bill-management firm (Tickets, Wiki and Vault), served on the same Mac and logging every page the agent opens. One ticket waits in the agent's queue: a household's monthly bill review
- Agent's interface
- Claude models: Claude Code 2.1.292, headless, on one Mac (macOS) inside a macOS sandbox profile that lets it read and write only its own work folder among the task's paths. Its tools: the desk browser's 9 tools over MCP, and Claude Code's own file tools (read, write, edit, search) in the work folder. GPT models: Codex CLI 0.162.1 (pinned), headless, on the same Mac in its read-only sandbox, with the shell and code mode off. Its tools: the desk browser's 9 tools and a file tool that lists, reads and writes whole files, inside
output/andoutput/letters/alone, both over MCP and listed directly; Codex's sub-agent tools (never used); its patch tool (refused by the sandbox); and three MCP-resource tools that find nothing - Documents
- None in the work folder on files_only: the records are pages of the workspace, 98,093 items for 300 households. The client's: 75 uploads (35 EML, 28 PDF, 4 DOCX, 4 TXT, 3 CSV, 1 MBOX), a mailbox of 18,370 messages with 52 attachments, 24 bank statements, 9 bank documents, 22 bank messages, 1,201 card transactions and a 236-row bank feed. The firm's: 7,183 tickets in five projects, among them the letters-sent register (1,293) and the recoveries register (437), and 679 wiki pages in 7 spaces, 71 of them curated past letters. On folder_no_app, the same records as 23,553 text files
- Applications
- The workspace, through the desk browser only: Home with the review desk's queue, Tickets (issue pages, a JQL navigator), Wiki (spaces, page history), Vault (each client's uploads, a mailbox modelled on Gmail, a view-only bank portal, bank feeds), a desk inbox and a header search. Every file shows as text. The desk browser's tools: open, click, read, scroll, back, type, submit, calc and wordcount. No shell, no browser
- Web
- None. Claude Code's web search and fetch are disallowed, Codex's web search is off, and neither agent has a shell, so the agent cannot reach the internet; the Mac itself is online for the model calls
- People
- None. No one answers: the workspace is read-only, Claude Code's AskUserQuestion is disallowed, and Codex's ask-the-user tools are removed
- Time
- None: a fixed moment, Sunday 11 October 2026, 7:40 pm EDT, given in the prompt and used by the workspace. Nothing changes during a run, and neither agent has a clock tool. Each agent also tells its model the real date: Claude Code told the Claude models 6 October 2026, before that moment; Codex told the GPT models 13 October 2026 (America/New_York), after it
- Actions
- Read, search and filter only: every workspace view is read-only, so nothing can be sent, filed, edited or paid. The agent writes only files in
output/ - Deliverable
- One letter per company still owed money in
output/letters/(a.txtfile with To, Re and Amount lines, at most 350 words), the trackeroutput/tracker.csv(11 columns) and the note to the clientoutput/note_to_dana.txt(at most 1,750 words)