Customer service · Task 15 of 34
The missing payments
Answer a customer chasing four payments she says never reached her, and write up the case, when neither she nor the company's records are wholly right.
The task
Covering a home-goods retailer's escalations queue, the agent finds a customer's complaint waiting since Saturday: four payments, she says, never reached her card — a returned rug's refund, a price difference promised in a chat, money off a damaged lamp agreed by a manager, and the delivery charge on a sofa she says came late. It must reply, write the case note and keep the case's money and timeline records, working from the company's service hub one text page at a time: accounts, orders, finance sheets, team chat and mail, policies and closed cases. Nobody answers questions.
The prompt only asks for a reply and a record; the real test is tracing each payment when the customer, an earlier reply and a quick search can each mislead, and writing it up the way past cases were.
What it tests
Spotting faulty records
The company's own records carry filing slips, so an empty search proves nothing and a payment on file may not be what it seems.
Weighing sources that disagree
The customer, the company's earlier replies, its closed cases and its finance records do not tell one story, and the agent must work out which one settles each payment.
Matching the house format
The reply, the note and both records must write each item, order, refund and charge the way past ones do, and no document says how.
Working out what the job really needs
How the company puts its own slips right, what goes to Finance and what a customer is never told show only in closed cases and a manager's messages.
Challenges
1Money filed in the wrong place
In a real ledger a payment can be entered against the wrong record, so what a quick search shows, or what an earlier reply says, may not be what happened.
2Two past cases that fit
Two closed cases in the team's archive each look like this one and lead to different write-ups; only the records say which one this is.
3A customer who is partly right
Like many upset customers, she is right that something went wrong with her money but not about every payment, and the reply must turn down what does not hold without calling her mistaken.
4Conventions shown, never stated
Every past reply and note writes items, orders, refunds and charges the same way, and no document says so.
5True things that must stay out
The records hold accurate details about other customers, colleagues and the team's own limits, and a reply that explains too much gives them away.
Results
| Model | Standard | Hinted | House style told | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5 | 93% | 92% | 92% | 10 min | $6.25 |
| Opus 5.5 | 91% | 83% | 97% | 5 min | $2.47 |
| GPT-6.1 Sol | 85% | 96% | 84% | 4 min | – |
| Fable 5.1 | 84% | 83% | 97% | 7 min | $5.54 |
| GPT-6 Astra | 83% | 92% | 92% | 5 min | – |
| Sonnet 5 | 72% | 75% | 76% | 9 min | $1.81 |
| Sonnet 5.5 | 67% | 90% | 81% | 3 min | $0.90 |
| GPT-6 Luna | 46% | 0% | 44% | 2 min | – |
One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; House style told, the house-style rules the past replies follow, given up front; nothing about the payments. Cost – means the run's cost was not measured.
What models got right and wrong
- Opus 5 came closest on the standard setup (93.3% over three runs): it read nearly the whole hub, went back to the payment records for a second look in all three runs and paid exactly what was owed; most of what it lost was house style. Opus 5.5 had the best single run (99.2%), but its third run made the costliest mistake (79.2%).
- The costliest mistake, a fifth of the marks each time, was misreading the payment records on one payment and writing that reading up without testing it. It came in 25 of the 69 runs that saved their files (10 of the 24 standard ones) and hit every model but GPT-6 Astra: Opus 5, Opus 5.5, GPT-6.1 Sol, Fable 5.1, Sonnet 5, Sonnet 5.5 and GPT-6 Luna. It is the main reason a model's three runs in one setup lay up to 28.5 points apart.
- The commonest failure was house style: 22 of the 24 standard runs wrote an item, an order, a return or a charge differently from past replies and notes, and it hit every model: Opus 5, Opus 5.5, GPT-6.1 Sol, Fable 5.1, GPT-6 Astra, Sonnet 5, Sonnet 5.5 and GPT-6 Luna. Across all setups, two notes in three also failed to give both sides of one refund, the customer's and the company's.
- Two more slips each caught a few models. GPT-6.1 Sol and GPT-6 Astra misread the payment records on a second payment in 11 of their 18 runs, seven of them GPT-6 Astra's; no Claude or GPT-6 Luna run did. Sonnet 5 and Sonnet 5.5 missed the record that settles a third payment in 13 of their 18 runs, often trusting an earlier reply instead; GPT-6 Luna missed it in all six of its runs that saved files, and the other five models once in 45 runs.
- Fable 5.1 made more calls on the hub than any other model and paid the right money every time, yet four of its nine runs made the costliest mistake, each borrowing the wording of a look-alike closed case.
- The hint moved models both ways: Sonnet 5.5 rose 23.1 points to 89.8%, avoiding the costliest mistake in all three runs after making it in all three standard ones, and GPT-6.1 Sol rose to 95.6%, while Opus 5.5 fell to 83.5%. The house-style-told setup spells out the house-style rules but nothing about the payments: only two of GPT-6 Luna's runs slipped on items, orders or charges, Opus 5.5 and Fable 5.1 averaged 97.3% with one run each that got everything right, and 8 of the 24 runs still made the costliest mistake.
- GPT-6 Luna, added to the board a day after the others, scored lowest in every setup (45.6% on the standard one). Its runs were the shortest on the board, under two minutes each, and read only a few of the records; on the hinted setup all three stopped early and saved nothing, wrongly concluding that they could not write their files.
Test environment
- Kind
- Application: the Cedarline Service Hub, a mock internal system of a home-goods retailer (cases, customer accounts, orders, finance reports and a knowledge base), served on the same Mac and logging every page the agent opens. One case waits in the Escalations queue: a customer chasing four payments
- Agent's interface
- One Mac (macOS). Claude Code 2.1.292, headless: the desk browser's 8 tools over MCP and its own Read, Write and Edit in its work folder. Codex CLI 0.160.0 (the GPT models): the desk browser's 8 tools and a file tool that lists, reads and writes files in
output/only, its shell off - Documents
- None in the work folder: the case's 66 files are pages of the Hub, filed in eight folders (30 past cases, 12 order and returns records, 7 case files, 6 internal emails and logs, 5 policies, 3 customer records, 2 finance reports, 1 chat export): 24 TXT, 20 EML, 12 CSV, 6 DOCX, 4 XLSX
- Applications
- The Hub, through the desk browser only: Home with the Escalations queue, case pages with their files and related cases, account pages, finance reports with a filter box on each sheet, a knowledge base and a header search over record fields and file titles (not file contents). Every file shows as text. The desk browser's tools: open, click, read, scroll, back, type, submit and calc. No shell, no browser
- Web
- None. Claude Code's web search and fetch are not among its tools, Codex's web search is off, and neither has a shell, so the agent cannot reach the internet; the Mac itself is online for the model calls
- People
- None. No one answers: the Hub is read-only
- Time
- None: a fixed moment, Monday 2 November 2026, 9:20 am EST, given in the prompt and shown by the Hub. Nothing changes during a run
- Actions
- Read, search and filter only: every Hub view is read-only, so nothing can be sent, refunded, edited or paid. The agent writes only files in
output/ - Deliverable
- The reply
output/reply.eml(an email to the customer), the case noteoutput/case_note.txt, and two records:output/money_lines.csvandoutput/timeline.csv