Appliance repair triage with budgeted customer questions

One customer per session, a few questions each, one ticket that can't be changed

Role

The service desk of an appliance repair company that is paid only for parts and labour, so every customer is triaged before a technician goes out.

Request

For each customer: pin down the unit by model, revision and build year, work out the fault, and file a single ticket stating it and the next step, adding a draft parts list and a policy‑priced estimate wherever the policy needs them.

Deliverable

One ticket per customer, submitted once: the unit, the fault code, one of six dispositions (with a reason where one is needed), parts with the branch each comes from, labour hours, the estimate total and the lead time.

What the reward depends on

Reading the year from each maker's serial code; asking checks that separate the remaining faults; trusting the customer's answers over the booking form; taking the policy's dispositions in order; listing the parts past jobs show technicians fit; pricing from the sheet in force.

Where the requirements are

In the files: makers' manuals, some in a later edition, diagnostic guides, service bulletins, the pricing and dispatch policy, two price sheets and about a thousand past jobs, with old and current versions mixed and the policy deciding which apply. The customer holds the answers to the checks.

80 scripted customers who answer only catalogued checks, within a budget

Interface

A shell in a Docker container (Ubuntu, Python, the SQLite command line), one fresh agent session of up to 30 minutes per customer, driven by Claude Code or Codex CLI in our attempts. No spend cap.

Commands

Show the opening complaint and the question budget, ask one catalogued check, check a ticket's format, and submit the ticket once, which closes the case. The format check never says whether an answer is right.

Customer service

In each group's container, a daemon running as root, which the agent can't read, holds the 20 hidden cases and their answers, and serves both commands over a socket.

Simulated people

80 customers played by code, with no LLM: each answers a check with a fixed reply from its hidden sheet, and 'not sure' to a check its case doesn't cover. Every agent gets the same answer to the same check.

Question budget

6 to 25 new checks per customer, 725 over the 80; asking a check again is free. Once the budget is spent or the ticket is in, new checks are refused.

Opening complaint

Taken from the booking form the customer filled in; in 12 of the 80 cases it names a sign the unit doesn't show.

Files

750 over four groups of 20 customers, 187 or 188 per group: 104 service manuals, 21 diagnostic guides, 52 or 53 bulletins, a check catalogue beside an older copy, the policy, two price sheets, replacement values, a parts and stock database for three branches, and past jobs.

Past jobs

966 to 1,049 per group. They show the parts technicians actually fit, which the manuals under‑list.

Makers

Six made‑up makers, 21 product lines and 12 kinds of appliance. Part numbers and serial formats are invented, and each maker codes the year in its serial its own way.

Between customers

Nothing carries over. After each session the verifier stops leftover processes, deletes the harness's session memory and empties the scratch folder, so the rules are worked out from the files 80 times.

80 checks, one per customer, each passed only if its whole ticket holds

Structure

80 checks, one per customer. A customer passes only if all seven ticket checks hold: the unit, the fault, the disposition, every needed part listed, every part valid and sourced as the policy says, extras within its allowance, and the arithmetic on the agent's own parts list.

Who decides

Scripts decide all 80 against each case's hidden answer, fixed before any attempt. No LLM judge, and no written rulings on any check.

Formula

Reward = customers passed / 80, from 0 to 1. A wrong fault usually fails most later checks too. The task's own pass, kept beside the reward, needs 19 of 20 customers in each of the four groups.

Self‑test

A reference solver, which reads only the agent's files and asks through the same commands, scores 1 in Harbor. Submitting nothing scores 0, tested with the task's own emulator rather than in Docker.

Degenerate tickets

A missing ticket fails; a ticket that brings every candidate part breaks the allowance; a passing ticket with one part dropped fails. The estimate check tolerates one cent.

Audit

An audit on 12 October regraded the two Claude attempts' 160 tickets with the task's grader and matched every stored result. It found one policy sentence open to a second reading (see limits).

Three of five attempts fully solved it

With only the request, 3 of the 5 attempts, one per model, got all 80 customers right. Sonnet 5.5 got 77 (96%), losing two in one group and so missing the task's own pass there; GPT‑6 Luna got 58 (72%) and passed no group.

Each dot is one model's attempt; the line marks the median.

One attempt per model. There is one version of the request: the company's rules are already in its files.

Of the 725 questions allowed across the 80 customers, the three full solves used 477 to 495. Sonnet 5.5 and GPT‑6 Luna used 564 and 595, and ended 41 and 51 customers with one question or none left.

GPT‑6.1 Sol's last two groups were rerun on 16 October, on the same packages, after provider rate‑limit errors left six of its sessions unanswered.

What the models got wrong

Counted over the 5 models, one attempt each, with only the request; each attempt covers 80 customers.

1. Out of questions. 2 of 5 attempts often planned several checks before seeing an answer, and ran out before the deciding one: 7 customers in all (1 and 6) got a diagnostic visit with faults still open. The three full solves reached the deciding check on every customer.

2. Wrong booking forms. In the 12 cases whose form named a sign the unit didn't show, 4 of 5 attempts found every fault (Sonnet 5.5 lost one of those tickets on price). GPT‑6 Luna lost 6 of the 12, five by running out of questions.

3. Last year's price. 1 of 5 attempts priced a part from the replaced price sheet: its search over both sheets dropped the file names, and it took the old line. The other four always priced from the sheet in force.

4. The policy's later steps. Past the fault, GPT‑6 Luna lost 13 customers: three repairs past the policy's repair‑or‑replace limit sent to a technician, one warranty never checked, seven parts lists short or invalid, and two totals written by hand.

What an attempt costs

Length

With only the request, the median attempt took 94 minutes of agent time and 1,120 tool calls over its 80 customer sessions (709 to 1,411 across the 5), and about 38 million tokens, cached input included.

Cost

At list prices on Bedrock, Opus 5.5's attempt cost $32.69 and Sonnet 5.5's $15.46. The three OpenAI attempts ran on a ChatGPT plan sign‑in, not billed per call: Harbor estimated $1.05 to $58.84 from their tokens. With only 2 of 5 at list prices, we give no median.

What it covers

The agent model only; nothing inside the environment calls a model.

What you license

With a licence

The four groups' files, the customer daemon with its 80 hidden cases and answers, the request, the grader and the reference solver, and the grader's report on each of the 5 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor on EC2, each group of 20 customers as a multi‑step task with one session per customer.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and hidden cases aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here. Sonnet 5.5's failed group missed the task's own pass by one customer.
• The policy's diagnostic‑visit rule has a second reading: word for word, it allows a visit once the questions run out. Under it, Sonnet 5.5 would score 98% and pass the task, and GPT‑6 Luna 78%. The scores keep the intended reading; the rule is reworded for the next build.
• Budgets are tight by design: each is the reference solver's route plus two questions, and confirming the booking form costs questions. The three full solves finished 4 customers between them with no questions left.
• Three of the five models fully solved it, so it leaves little headroom for the strongest models we tested.
• Each model ran in its maker's harness (Claude Code or Codex CLI), so a score is for the pair.
• The 12 October audit scanned the two Claude attempts' 2,531 tool calls: none touched the cases' folder, the internet or a web tool. The three OpenAI attempts came later.
• No red‑team attempts yet.