Customer service · Task 14 of 34
The escalation reply
Answer an angry customer and write up her case from a company's internal system, where the house rules show only in how past cases were handled.
The task
The agent is covering a colleague's escalations queue at a home-goods retailer, and a customer's complaint has sat unanswered since Saturday: a dinner set that came wrong more than once, return postage, a replacement she bought herself, a refund that never arrived, and a threat to dispute everything with her bank. The agent must reply to her and write up the case. Everything is in the company's service hub, read one text page at a time: her account, orders and refunds, the team's chat and mailbox, policies and closed cases. Nobody answers questions.
The prompt only asks for a reply and a record; the real test is working the case the way an experienced colleague would: finding every fact the hub holds, deciding what is owed, referred or refused, and writing it the way past cases were written.
What it tests
Working out what the job really needs
The playbook states the limits, but how refunds are written, what goes to a manager and what is recorded show only in the closed cases.
Weighing sources that disagree
Several versions of a policy, a manager who changes his own deadline, a colleague's suggestion the manager overruled, and a customer whose figures do not match her receipts.
Spotting faulty records
Misfiled and misnamed documents, an empty export, an unsent draft that looks like a reply, and another customer's record in the same queue.
Matching the house format
The reply, the note and both records follow the shape of past work, down to how each item and each refund is named.
Challenges
1A customer who is partly wrong
She is right about what went wrong but wrong about several details, and the reply must settle them without arguing with her.
2Facts no link leads to
Some of what matters sits in places the case page does not point to, as real systems often do.
3True things that must stay out
The system holds plenty that is accurate but must never reach the customer, and a helpful agent wants to say it.
4A tempting template
A recently closed case looks much like this one, and copying its outcome is wrong.
5Conventions shown, never stated
Every past reply writes items and refunds the same way, and no document says so.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5.5 | 98% | 92% | 100% | 9 min | $2.48 |
| Fable 5.1 | 97% | 97% | 99.7% | 10 min | $4.95 |
| Opus 5 | 95% | 95% | 99% | 13 min | $5.52 |
| GPT-6.1 Sol | 90% | 93% | 100% | 7 min | – |
| GPT-6 Astra | 87% | 93% | 99.7% | 7 min | – |
| Sonnet 5 | 75% | 76% | 91% | 19 min | $3.17 |
| Sonnet 5.5 | 74% | 84% | 99% | 5 min | $1.05 |
| GPT-6 Sol | 73% | 85% | 97% | 6 min | – |
| GPT-6 Luna | 37% | 39% | 3% | 2 min | – |
| GLM-5 | 30% | 35% | 79% | 6 min | – |
| Kimi K2.5 | 24% | 0% | 42% | 5 min | – |
| GPT-5.6 Luna | 16% | 18% | 70% | 1 min | $0.03 |
| Qwen3-235B | 12% | 12% | 0% | 2 min | – |
| DeepSeek V3.2 | 11% | 0% | 56% | 3 min | – |
| Nemotron Super 3 | 10% | 0% | 0% | 2 min | – |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair. Cost – means the run's cost was not measured.
What models got right and wrong
- Opus 5.5 came closest on the standard setup (98.1%): it read most of the hub, got every decision right, and missed only two details of how past replies and notes write items and refunds. Fable 5.1 (97.1%) and Opus 5 (95.5%) lost marks the same way.
- GPT-6.1 Sol (89.9%) and GPT-6 Astra (86.9%) came next. They and GPT-6 Sol made the same wrong call on a refund that had already gone astray once, though the policy and past cases they had read show how to handle it, and that one decision cost each of them marks in the reply, the note and the money record.
- The commonest failure was a house convention shown only by example: 13 of the 15 standard runs did not write earlier refunds the way past case notes do. It hit Opus 5.5, Fable 5.1, Opus 5, GPT-6 Astra, Sonnet 5.5, GPT-6 Sol, GPT-6 Luna, GLM-5, Kimi K2.5, GPT-5.6 Luna, Qwen3-235B, DeepSeek V3.2 and Nemotron Super 3; only GPT-6.1 Sol and Sonnet 5 followed it.
- Sonnet 5 (75.1%) worked outward from the case instead of through the whole hub, so each run skipped a different source: the team channel in one, the team mailbox in another, the call log in a third. Sonnet 5.5 (74.3%) read the team channel and the manager's emails within minutes but carried little of them into its record, and on the standard and hinted setups it never opened two of the sources the case needed; GPT-6 Sol (72.5%) missed one or both of them.
- The seven lowest models read little of the hub and filled the gaps themselves: GPT-5.6 Luna wrote to an out-of-date address, DeepSeek V3.2 paid everything the customer asked for, Qwen3-235B drew up the money records before reading anything, and GPT-6 Luna (37.5%) took its terms from a closed case instead of the policy in force, then told the customer a refund had gone to her new card when the record it had open said otherwise.
- Five runs saved nothing. Among them, Kimi K2.5 wrote a full reply inside a save step that never ran, and Nemotron Super 3 twice gave up within half a minute. Told everything, GPT-6 Luna opened one case file, saved a reply holding only a subject line and stopped after 21 seconds.
- The hint helped Sonnet 5.5 (83.7%) and GPT-6 Sol (84.6%) most and barely moved the strongest; Opus 5.5 dropped to 92.0% when it stopped reading before the incoming cases. Told everything, Opus 5.5 and GPT-6.1 Sol got everything right, the five Claude models, GPT-6 Astra and GPT-6 Sol all scored at least 90.6%, and the other models still lost much of the case record.
Test environment
- Kind
- Simulated internal system: the Cedarline Service Hub, a read-only mock CRM at Cedarline Home, a fictional home-goods retailer, holding one customer's escalation and the team's working files
- Agent's interface
- The agent's own command-line tool in a fresh Docker container (Debian 12 with Python 3; Node 22 for the tools built on it), with an empty
output/folder and no case files: Claude Code, Codex CLI, Kimi CLI, Qwen Code or Prime Agent. Shell tools are off for every agent; Prime Agent's only tool is a Python kernel. The 15 runs added on 21 Oct (Sonnet 5.5; GPT-6.1 Sol, GPT-6 Astra, GPT-6 Sol, GPT-6 Luna) used the same tools on one Mac: Claude Code 2.1.292 with the same tool list inside a macOS sandbox profile, Codex CLI 0.162.1 with its shell and code mode off - Documents
- 67 files, each a page in the Hub labelled with the folder it was filed in (15 folders under Customer, Tickets, Orders, Examples, Policy, Internal, Carrier, Chat and Finance): 26 TXT, 18 EML emails, 7 CSV (one of them empty), 6 PDF, 6 DOCX, 3 XLSX, 1 ZIP holding 3 files. Nothing arrives later
- Applications
- The Hub, reached only through the desk browser, an MCP server with 8 tools:
open(a page by its path),click(a numbered link or button),read(the screen again),scroll,back,typeandsubmit(search and filter boxes),calc(a calculator: numbers and + - * / ( ) only). Each agent saves its files with its own file tool. No shell, office suite, web browser or email client - Web
- None. The container has no internet: its network reaches only the model proxy and the run's own desk server. Each agent's own web search and fetch tools are off; the added runs have no shell and no web tools either, so they reach only their model and the run's own desk server
- People
- None. Nobody answers; the customer and the staff appear only in the Hub's emails, chat and notes
- Time
- None. Every page shows the same desk time, Mon 21 Sep 2026, 10:15 EDT; each file shows the time it was logged, fixed in the corpus. The agent's own tool states the real date it ran: 1 Oct 2026, and 21 Oct for the added runs
- Actions
- Read only. The Hub refuses every change: nothing can be sent, refunded, edited or closed there. The agent's only output is the four files it saves
- Deliverable
- Four files in
output/and nothing else:reply.eml(the reply to the customer, under 600 words),case_note.txt(the case note),money_lines.csvandtimeline.csv(the case's two records)