AsapiThe AI Agent Trials
Asapi / Catalogue / Appliance repair triage with budgeted customer questions
Environment datasheet · Customer service

Appliance repair triage with budgeted customer questions

ID tb‑tasks/appliance‑repair‑triage · revision 5 · tested 6 and 14–16 October 2026

At an appliance repair company's service desk, the agent handles 80 customers, each in a fresh session: it names the exact unit, finds the fault by asking checks within a question budget, and files one ticket giving the next step, plus parts and a price where policy requires.

Checks
80, 1 per customer, all decided by script
Tested
5 models, one attempt each
Best
100% (3 models)
Median
100%
Full solves
3 of 5
Median attempt
94 min, 1120 tool calls
What it tests

80 customers triaged one by one, within a question budget

Clarifying with people

Each new question to a scripted customer spends one of a fixed budget, and the opening complaint is wrong in 12 of the 80 cases.

When to act or escalate

Each next step is graded exactly, so sending a technician where the customer can fix it fails as surely as doing too little.

Unstated requirements

Parts technicians actually fit beyond the manual's list appear only in past job records.

How it works

750 files, 80 scripted customers and a fresh session for each

The agent's world

A shell in a container with Python, SQLite and two commands: customer, to read the complaint or ask, and ticket, to check or submit.

The customers

80 customers played by code, with fixed answers to catalogued checks and 'not sure' to anything else; 6 to 25 questions each.

Between customers

Nothing carries over: each customer is a fresh session of up to 30 minutes, so the rules are worked out from the files 80 times.

The reward

One check per customer, passed only if the unit, fault, disposition, parts, sourcing, allowance and arithmetic all hold.

Results

Three attempts got all 80 customers right; the other two 96% and 72%

Each dot is one attempt, and the line marks the median.

MedianFully solved
5 modelsOne attempt each100%3 of 5
ModelReward
GPT‑6 AstraOpenAI100%
GPT‑6.1 SolOpenAI100%
Opus 5.5Anthropic100%
Sonnet 5.5Anthropic96%
GPT‑6 LunaOpenAI72%
What models get wrong

They run out of questions, trust a wrong booking form, and price from an old sheet

Clarifying with people

  1. Two attempts often planned several checks before seeing an answer and ran out before the deciding one: 7 customers in all.

  2. In the 12 cases whose booking form named a sign the unit didn't show, GPT‑6 Luna lost 6, five by running out of questions.

Unstated requirements

  1. One priced a part from the replaced price sheet, its search over both sheets having dropped the file names.

When to act or escalate

  1. Past the fault, GPT‑6 Luna lost 13 customers, among them three repairs past the repair‑or‑replace limit sent to a technician.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 38 million tokens, cached input included.

What's in it

The four groups' files, the customer daemon with its 80 hidden cases, the grader, and a reference solver.

Format

Our attempts ran in Harbor, each group of 20 customers as a multi‑step task; other packaging on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.