Appliance repair triage with budgeted customer questions
ID tb‑tasks/appliance‑repair‑triage · revision 5 · tested 6 and 14–16 October 2026
At an appliance repair company's service desk, the agent handles 80 customers, each in a fresh session: it names the exact unit, finds the fault by asking checks within a question budget, and files one ticket giving the next step, plus parts and a price where policy requires.
- Checks
- 80, 1 per customer, all decided by script
- Tested
- 5 models, one attempt each
- Best
- 100% (3 models)
- Median
- 100%
- Full solves
- 3 of 5
- Median attempt
- 94 min, 1120 tool calls
80 customers triaged one by one, within a question budget
Clarifying with people
Each new question to a scripted customer spends one of a fixed budget, and the opening complaint is wrong in 12 of the 80 cases.
When to act or escalate
Each next step is graded exactly, so sending a technician where the customer can fix it fails as surely as doing too little.
Unstated requirements
Parts technicians actually fit beyond the manual's list appear only in past job records.
750 files, 80 scripted customers and a fresh session for each
The agent's world
A shell in a container with Python, SQLite and two commands: customer, to read the complaint or ask, and ticket, to check or submit.
The customers
80 customers played by code, with fixed answers to catalogued checks and 'not sure' to anything else; 6 to 25 questions each.
Between customers
Nothing carries over: each customer is a fresh session of up to 30 minutes, so the rules are worked out from the files 80 times.
The reward
One check per customer, passed only if the unit, fault, disposition, parts, sourcing, allowance and arithmetic all hold.
Three attempts got all 80 customers right; the other two 96% and 72%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 100% |
| GPT‑6.1 SolOpenAI | 100% |
| Opus 5.5Anthropic | 100% |
| Sonnet 5.5Anthropic | 96% |
| GPT‑6 LunaOpenAI | 72% |
They run out of questions, trust a wrong booking form, and price from an old sheet
Clarifying with people
Two attempts often planned several checks before seeing an answer and ran out before the deciding one: 7 customers in all.
In the 12 cases whose booking form named a sign the unit didn't show, GPT‑6 Luna lost 6, five by running out of questions.
Unstated requirements
One priced a part from the replaced price sheet, its search over both sheets having dropped the file names.
When to act or escalate
Past the fault, GPT‑6 Luna lost 13 customers, among them three repairs past the repair‑or‑replace limit sent to a technician.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 38 million tokens, cached input included.
What's in it
The four groups' files, the customer daemon with its 80 hidden cases, the grader, and a reference solver.
Format
Our attempts ran in Harbor, each group of 20 customers as a multi‑step task; other packaging on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials