Customer service · Task 20 of 34
The repair desk's triage
Triage 80 appliance-repair customers one fresh session at a time: name the exact unit, find the fault on a question budget, and quote by the company's policy.
The task
A made-up appliance repair company is paid only for parts and labour, so its service desk triages every customer before a technician goes out. The agent takes 20 customers, one per fresh session: it reads the customer's booking form, asks checks from a catalogue within a question budget, and submits a ticket naming the exact unit, the fault, what happens next, the parts to bring and the price. It has the makers' manuals, diagnostic guides, service bulletins, the company's policy, a parts and stock database and a thousand past jobs.
The prompt only asks for a correct ticket. The real test is careful desk work on every customer: trusting the customer's answers over the form, spending questions where they separate faults, and using only the documents that are in force.
What it tests
Asking the right questions
Each customer allows only a few checks, so each one has to narrow the faults that are left.
Knowing which document wins
Manual editions, bulletins and price sheets come in old and new versions side by side.
Spotting bad data
The booking form is sometimes wrong, and only the customer's own answers count.
What the job really needs
The parts technicians actually fit show up in past jobs, not in the manual alone.
Challenges
1A form that can be wrong
Customers fill in the booking form before the call, as in real life, and sometimes describe the wrong sign.
2Old papers still on the shelf
Superseded bulletins, an older check catalogue and last year's price sheet sit beside the current ones.
3A limited number of questions
A check that cannot tell the remaining faults apart wastes part of a small budget.
4Every maker its own serial code
The year of manufacture has to be read the way that maker writes it.
5No memory between customers
Each session starts fresh, so every customer is worked out from the files again.
Results
| Model | Standard | Time (standard) | Cost (standard) |
|---|---|---|---|
| GPT-6 Astra | 100% | 94 min | $58.84 |
| GPT-6.1 Sol | 100% | 81 min | $10.48 |
| Opus 5.5 | 100% | 202 min | $32.69 |
| Sonnet 5.5 | 96% | 48 min | $15.46 |
| GPT-6 Luna | 72% | 108 min | $1.05 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked.
What models got right and wrong
- GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 passed all 80 customers (100.0%): all three read the documents before asking anything, read past jobs before every parts list and priced only from the sheet in force. GPT-6.1 Sol also asked how the appliance was used when past repairs depended on it.
- The commonest failure was running out of questions before the one that settled the fault, then sending a technician to diagnose: it hit Sonnet 5.5 once and GPT-6 Luna six times. Both planned several checks before seeing the answers; the three models that passed everything also finished four customers between them with no questions left.
- Sonnet 5.5 passed 77 of 80 (96.2%) but failed the task, because two of its misses fell in the same group of 20 customers, which needs 19. Its other slips were a part priced from the replaced price sheet and the purchase year written as the year of manufacture.
- GPT-6 Luna passed 58 of 80 (72.5%) and no group of 20. Beyond the question budget it lost customers on every later step of the policy: repairs too costly to be worth doing sent to a technician, past-job extras dropped because it read the extras allowance as a limit on them, part changes for certain build years missed, and totals written by hand without a calculation.
- Among the three that passed everything, GPT-6.1 Sol was the quickest and cheapest (80.7 min, $10.48); Opus 5.5 took 202.0 min, and GPT-6 Astra cost the most, at about $59.
- There is no hinted or told-everything setup for this task: every score is on the job as asked.
Test environment
- Kind
- Simulated service desk: the triage desk of an appliance repair company, with made-up makers and models (Arvana, Corvo, Halden, Norvik, Pellam, Tovell; 21 product lines, 12 kinds of appliance), and customers played by a hidden daemon in the container. 4 variants (seeds 0 to 3) of 20 customers each, each variant its own run
- Agent's interface
- A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, the SQLite command line; 2 CPUs, 4 GB), one fresh session per customer as an unprivileged user, driven by the agent's own command-line tool through harbor: Claude Code or Codex CLI
- Documents
- Per variant, 187 or 188 files in
/app: indocs/, 104 Markdown service manuals (one per model revision, some with a later edition), 21 Markdown diagnostic guides (one per product line), 52 or 53 Markdown service bulletins, the check catalogue and an older copy of it, and a short index; inpolicy/, the pricing and dispatch policy (Markdown) and 3 CSV files (the price sheet in force, an older price sheet, replacement values);inventory.db(SQLite: parts, fits, kits, alternates, stock at three branches);history/jobs.csv(966 to 1,049 past jobs); and an emptywork/folder. 750 files over the 4 variants - Applications
- Two commands.
customer:complaint(the opening complaint, from the customer's booking form, and the question budget) andask CHECK(ask the customer one check from the catalogue).ticket:check FILE(a ticket's format) andsubmit FILE(submit the ticket, once; it closes the customer). The SQLite command line and Python read the files. No other apps - Web
- Not needed: nothing the task asks about is online. The container can reach the internet; the makers and models are made up. Claude Code's web page reader (WebFetch) was on for Opus 5.5 and Sonnet 5.5, and neither run used it; Codex's web search was turned off for GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna
- People
- 20 customers per variant, one per session, played by the daemon's code with no language model: each answers a check with a fixed reply from its case ("not sure" for a check the case does not cover), the same for every agent. The opening complaint can be wrong. Nobody else
- Time
- None. The documents and price sheets are dated, and the policy is in force from a stated date
- Actions
- Ask checks, within each customer's question budget (6 to 25 questions; asking a check again is free), and submit one ticket per customer. Nothing in the files can be changed
- Deliverable
- One JSON ticket per customer, submitted with
ticket submit: model, revision, year of manufacture, fault code, disposition (and its reason where one is needed), a draft parts list (part number, quantity, source branch), and labour hours, estimate total and lead time where the disposition needs a parts list