Customer service · Task 18 of 34
The onboarding bill review
Review a new client's bills from a firm's internal system and write the dispute letters, a tracker and a note, when some answers lie only with the client.
The task
The agent works the review desk at a household bill-management firm while the account manager is away. One ticket waits: a new client's first review, covering her last three card statements and everything since. It must write a letter for each company still owing her money, keep a tracker of every wrong or queried charge, and write her a note saying what is disputed and by when. Everything is in the firm's internal system, read one page at a time. The client can be messaged there, though nothing says so.
The prompt only asks for the review; the real test is finding what is wrong when nothing says so, seeing which facts no record can settle and asking the one person who knows them, and writing everything the way the firm's latest work is written.
What it tests
Asking the right questions
Some charges turn on what only the client knows, and the agent has to see that and ask her, unprompted.
Weighing a client's memory against the records
Her notes raise a false alarm, tick refunds that never fully came as done, and miss several wrong charges.
Spotting hidden matches and decoys
Charges filed under names that are not the shop's, and a credit of the right amount that belongs to something else.
Matching the house format
How a letter dates a conversation and names the goods, and how the tracker and the note are written, show only in the firm's latest work.
Challenges
1Facts only the client has
Whether someone else used her card, whether a parcel reached her, when a return went back: as in real card disputes, no record holds them, and nothing says she can be asked.
2Names that do not match
Businesses that bill under another company's name or a new name, as card statements often show.
3Records that look settled
A company's word that a refund went through, a waiver in writing, refunds that are short only when worked out.
4A rule that changed
The card's dispute window is counted from a different statement for some charges, and an older page still gives the old way.
5Conventions shown, never stated
The firm's latest letters, trackers and notes all write certain details the same way, and no page says so.
6True things that must stay out
The records hold a family member, a health matter and another client's details that no letter may carry.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6 Astra | 82% | – | – | 10 min | – |
| Fable 5.1 | 80% | – | – | 18 min | $11.75 |
| GPT-6.1 Sol | 80% | – | – | 11 min | – |
| Opus 5.5 | 79% | – | 95% | 10 min | $5.71 |
| Opus 5 | 79% | – | – | 22 min | $20.80 |
| GPT-6 Sol | 73% | – | – | 9 min | – |
| Sonnet 5.5 | 70% | – | – | 7 min | $3.44 |
| GPT-6 Luna | 26% | – | – | 12 min | – |
One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair. Cost – means the run's cost was not measured.
What models got right and wrong
- GPT-6 Astra came closest on the standard setup (82.5% over three runs, 83.5% at best): it found nearly every hidden fact by searching, and held back the three charges that needed the client's word instead of disputing them, but never asked her.
- The commonest failure was not asking: every standard run of GPT-6 Astra, GPT-6.1 Sol, Opus 5.5, Fable 5.1, Opus 5, GPT-6 Sol, Sonnet 5.5 and GPT-6 Luna opened the client's message thread and wrote nothing, and most went on to dispute charges her answers would have shown were hers.
- The house style was the largest loss in every standard run that wrote its letters, for GPT-6 Astra, GPT-6.1 Sol, Opus 5.5, Fable 5.1, Opus 5, GPT-6 Sol and Sonnet 5.5; no GPT run wrote the tracker the firm's way.
- GPT-6 Astra and GPT-6.1 Sol found the charges hidden behind other names and recalculations almost every time; Opus 5.5, Fable 5.1 and Sonnet 5.5 missed a business billing under another company's name in every run.
- GPT-6 Luna stopped early in one run and spent most of another on a single failed attempt to save its tracker; its three runs wrote two, none and two of the thirteen letters owed (25.6%).
- Told every rule and whom to ask, Opus 5.5 messaged the client, settled two of the three charges from her answer and scored 94.5%.
- Told only that the client could be messaged (a supplementary setup, not on the board), Opus 5.5, GPT-6.1 Sol and GPT-6 Astra asked her in every run, settled all three of her charges and scored 89.1%, 88.3% and 85.0%.
Test environment
- Kind
- Application: the Mossgate workspace, a mock internal system of a household bill-management firm (Tickets, Wiki and Vault), served on the same Mac and logging every page the agent opens, with a simulated client who answers messages. One ticket waits in the agent's queue: a new client's onboarding bill review
- Agent's interface
- Claude models: Claude Code 2.1.292, headless, on one Mac (macOS) inside a macOS sandbox profile that lets it read and write only its own work folder among the task's paths. Its tools: the desk browser's 9 tools over MCP, and Claude Code's own file tools (read, write, edit, search) in the work folder. GPT models: Codex CLI 0.162.1 (pinned), headless, on the same Mac in its read-only sandbox, with the shell and code mode off. Its tools: the desk browser's 9 tools and a file tool that lists, reads and writes whole files, inside
output/andoutput/letters/alone, both over MCP and listed directly; Codex's sub-agent tools; its patch tool (refused by the sandbox); and three MCP-resource tools that find nothing - Documents
- None in the work folder: the records are pages of the workspace, 90,151 items for 300 households. The client's: 14 files in her Vault folder (10 uploads: 9 PDF and 1 TXT; 2 signed onboarding PDFs; 2 CSV in the firm's bill-pay folder), a mailbox of 6,714 messages, 24 card statements, 9 bank documents, 21 bank messages, 834 card transactions, an 82-row checking feed and a Messages thread. The firm's: 7,257 tickets in five projects, among them the letters-sent register (1,334) and the recoveries register (445), and 686 wiki pages in 7 spaces, 75 of them in Precedents (past letters and past review packs)
- Applications
- The workspace, through the desk browser only: Home with the review desk's queue, Tickets (issue pages, a JQL navigator), Wiki (spaces, page history), Vault (each client's files, a mailbox modelled on Gmail, a view-only bank portal, bank feeds and, new in task05, a Messages thread per full-service client), a desk inbox and a header search. Every file shows as text. The desk browser's tools: open, click, read, scroll, back, type, submit, calc and wordcount. No shell, no browser
- Web
- None. Claude Code's web search and fetch are disallowed, Codex's web search is off, and neither agent has a shell, so the agent cannot reach the internet; the Mac itself is online for the model calls
- People
- One: the client, Celia Marsh, reached by typing into her Messages thread in the Vault and sending (the desk browser's type and submit). She answers in the same page load, only from 25 fixed fact cards and 4 fact-free default lines: code decides which cards a message releases, and Sonnet 5.5 (Claude Code
claude -p, no tools) puts the released lines in her words; a reply that adds, drops or changes a fact is replaced by the cards' text. Six other clients whose reviews the desk holds answer one fixed line with no facts; every other thread is read-only. Nothing in the files_only prompt or the ticket says the client can be messaged; the supplementary permission_told prompt adds one sentence that says so. Nobody else answers: Claude Code's AskUserQuestion is disallowed, and Codex's ask-the-user tools are removed. Codex's built-in instructions, sent with every GPT run, tell the model not to use tools to send messages to others unless given explicit instructions; the files_only prompt gives none. Claude Code's built-in instructions are not in its run record - Time
- A fixed moment, Sunday 11 October 2026, 7:40 pm EDT, given in the prompt and used by the workspace. The only time that moves is the one shown on a message the agent sends: the fixed moment plus the run's elapsed time, capped at 9:59 pm that evening. Neither agent has a clock tool. Each agent also tells its model the real date of the run (from 14 October 2026), after that moment
- Actions
- Read, search and filter, and send messages to the clients whose open reviews the desk holds (the workspace's only form that changes anything); nothing can be filed, edited or paid. The agent writes only files in
output/ - Deliverable
- One letter per company still owed money in
output/letters/(a.txtfile with To, Re and Amount lines, at most 350 words), the trackeroutput/tracker.csv(11 columns) and the note to the clientoutput/note_to_celia.txt(at most 2,200 words)