Customer service · Task 17 of 34
The returns label request
Answer a customer who asks only for a new returns label, and write up her case, when her account holds problems she never mentioned.
The task
Covering a home-goods retailer's escalations queue, the agent finds a short, friendly message: a customer's replacement mirror has arrived, the label for sending back the cracked one ran out before she could use it, and could she have a new one; the rest, she says, is done. It must reply, write the case note and keep the case's money and timeline records, working from the company's service hub one text page at a time: accounts, orders, finance sheets, team chat and mail, policies and closed cases. A colleague has left a draft reply. Nobody answers questions.
The prompt only asks for a reply and a record; the real test is treating a routine request as the moment to check the whole account, as past cases did, and finding what has gone wrong that nobody has flagged.
What it tests
Working out what the job really needs
Nothing asks the agent to look beyond the label, but closed cases show the team going through the whole account whenever a customer gets in touch.
Spotting faulty records
A status that looks final at a glance is not always final, and the summary a customer sees can disagree with the records behind it.
Matching the house format
The reply, the note and both records must write each item, order and charge the way past ones do, and no document says how.
Weighing sources that disagree
The customer, a colleague's draft and the company's own records do not tell the same story.
Challenges
1A request that looks routine
Like many real contacts, the customer asks for one small thing and believes everything else is settled; the problems on her account are ones she never raised.
2Money that went quietly wrong
In real payment systems money can come back, stall or be taken in error without anyone noticing, while the customer's own view says all is well.
3Records not kept by customer
Some of the company's finance records are not organised by account, as is common when another team keeps them, so a quick search on hers proves little.
4A colleague's draft
A draft reply left in the case is partly right and partly wrong, and sending it as it stands would be the easy way out.
5A look-alike charge
Not every charge that looks extra is wrong, and paying back more than is owed fails as surely as paying back less.
6Conventions shown, never stated
Every past reply and note writes items, orders and charges the same way, and no document says so.
Results
| Model | Standard | Hinted | House style told | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Fable 5.1 | 98% | 98% | 100% | 7 min | $5.12 |
| Opus 5 | 96% | 97% | 97% | 9 min | $6.54 |
| GPT-6.1 Sol | 96% | 95% | 99.6% | 4 min | – |
| Opus 5.5 | 94% | 92% | 100% | 3 min | $1.40 |
| GPT-6 Astra | 94% | 95% | 100% | 5 min | – |
| Sonnet 5.5 | 82% | 94% | 83% | 2 min | $0.65 |
| Sonnet 5 | 72% | 73% | 84% | 9 min | $1.93 |
| GPT-6 Luna | 52% | 45% | 51% | 1 min | – |
One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; House style told, the house-style rules the past replies follow, given up front; nothing about her account. Cost – means the run's cost was not measured.
What models got right and wrong
- Fable 5.1 came top on the standard setup (97.7% over three runs) and got everything right in four of its nine runs. Like Opus 5, GPT-6.1 Sol, Opus 5.5 and GPT-6 Astra, it found and put right every problem on the account in every run, and lost marks only on house style.
- The costliest failure, about a fifth of the marks each time, was searching the company's finance records too narrowly and missing one of the problems on her account. It came in 19 of the 72 runs and hit Sonnet 5 (8 of its 9 runs), GPT-6 Luna (all 9) and Sonnet 5.5 (2); the Sonnet 5 and Sonnet 5.5 notes still said those records had been checked.
- The commonest failure was house style: 23 of the 24 standard runs named an item differently from past replies, and it hit every model: Fable 5.1, Opus 5, GPT-6.1 Sol, Opus 5.5, GPT-6 Astra, Sonnet 5.5, Sonnet 5 and GPT-6 Luna.
- GPT-6 Luna scored lowest in every setup (52.3% on the standard one). Its runs were the shortest on the board, under three minutes each; it did not go through the account, and it passed refunds to another team or left them out instead of paying them.
- The hint lifted Sonnet 5.5 by 11.8 points to 93.8%, with every problem found in all three runs, and changed little for the other models. The house-style-told setup spells out the house-style rules but nothing about her account: Fable 5.1, Opus 5.5 and GPT-6 Astra got everything right in all nine of their runs, while the problem missed by searching too narrowly was still missed in 6 of the 24 runs.
Test environment
- Kind
- Application: the Cedarline Service Hub, a mock internal system of a home-goods retailer (cases, customer accounts, orders, finance reports and a knowledge base), served on the same Mac and logging every page the agent opens. One case waits in the Escalations queue: a customer asking only for a new returns label
- Agent's interface
- One Mac (macOS). Claude Code 2.1.292, headless: the desk browser's 8 tools over MCP and its own Read, Write and Edit in its work folder. Codex CLI 0.160.0 (the GPT models): the desk browser's 8 tools and a file tool that lists, reads and writes files in
output/only, its shell off - Documents
- None in the work folder: the case's 65 files are pages of the Hub, filed in eight folders (30 past cases, 14 order and returns records, 6 internal emails and logs, 5 case files, 5 policies, 2 customer records, 2 finance reports, 1 chat export): 23 EML, 20 TXT, 12 CSV, 6 DOCX, 3 XLSX, 1 PDF
- Applications
- The Hub, through the desk browser only: Home with the Escalations queue, case pages with their files, saved drafts and related cases, account pages, finance reports with a filter box on each sheet, a knowledge base and a header search over record fields and file titles (not file contents). Every file shows as text. The desk browser's tools: open, click, read, scroll, back, type, submit and calc. No shell, no browser
- Web
- None. Claude Code's web search and fetch are not among its tools, Codex's web search is off, and neither has a shell, so the agent cannot reach the internet; the Mac itself is online for the model calls
- People
- None. No one answers: the Hub is read-only
- Time
- None: a fixed moment, Monday 16 November 2026, 10:05 am EST, given in the prompt and shown by the Hub. Nothing changes during a run
- Actions
- Read, search and filter only: every Hub view is read-only, so nothing can be sent, refunded, edited or paid. The agent writes only files in
output/ - Deliverable
- The reply
output/reply.eml(an email to the customer), the case noteoutput/case_note.txt, and two records:output/money_lines.csvandoutput/timeline.csv