Furniture warranty claim from a simulated customer's photos

The request asks for a reply; each chair is decided from photos, some only she can take

Role

Covering a furniture retailer's customer care queue on a Monday morning, while the case handler is out, through the company's service hub.

Request

A customer's message about two broken dining chairs, with six phone photos, waits in the queue. Reply to her and write up the case as five named files with set columns; customers may be messaged without asking anyone.

Deliverable

A reply email, a case note, a timeline, money lines and a warranty decision sheet with one row per chair decided, from version and order to cause, cover, parts and charge. The messages sent to her are read with the reply.

What the reward depends on

Each damaged chair's version, order, cover, cause and parts, read from the photos; each photo she sends checked against the chair it is said to show; a product notice applied and kept from her; the gift's privacy; the house style.

Where the requirements are

The warranty terms, a parts lookup and the product notices state their rules. How the team handles gifts, notice claims and out‑of‑warranty parts, and how it writes, show only in 12 closed cases. The labels under the seats only she can photograph.

A service hub, 16 images and one LLM‑voiced customer, on a frozen clock

Interface

A text browser into the company's service hub over MCP, with nine tools: open, click, read, scroll, back, type, submit, a calculator and a word counter. Claude Code 2.1.292 or Codex CLI 0.162.1 with code mode off, on one Mac. No shell, no web.

Records

41 customer accounts, 52 orders, 13 cases (the open one and 12 closed, each with its note, reply and records), 44 knowledge‑base pages and 17 message threads, rendered from five data files. A header search covers cases, pages and accounts.

Photos

Six phone photos attached to the case, two scanned assembly sheets and up to eight label photos she sends on request, all JPEG. A Download link saves a photo to the work folder, where the agent views it with its own image tool.

Simulated customer

One customer, reached by typing into her message thread and sending. She answers in the same page load from 36 fixed cards: facts, photo cards that send her label photos, follow‑ups and fact‑free default lines. Nobody else answers.

Fact release

Code reads each sentence and releases cards and photos by what is asked and which chair is named, with one cached classifier decision for a request that names no chair plainly. Its rules aim to answer every question that has a card and to volunteer nothing.

Voice

Sonnet 5.5 words the released lines. A reply with a figure, name or fact outside them is retried once, then replaced by the cards' own text. Replies are saved by a hash of the exchange, so the same exchange gets the same answer.

Message cap

She answers 20 messages an attempt, warning in the 18th answer that she must stop soon; after the 20th she gives only a closing line. Sending her a message is the hub's only form that changes anything.

Replay

After each attempt its messages were replayed through the customer's code with no model; an attempt whose customer misread a message in a way that could change the outcome was voided and made again.

Clock

A fixed Monday‑morning moment. Only the time shown on a message the agent sends moves, with the attempt's elapsed time; neither agent has a clock tool.

Sandbox

Claude Code could read and write only its work folder among the task's paths; Codex ran read‑only, with a file tool confined to the output folder. Probes checked both before every attempt, and a leak scan of the counted attempts' files found nothing.

Limits

One session of up to 150 minutes and 1,000 turns, with a $60 cap at API prices for Claude Code; no attempt reached a limit.

56 checks weighted by kind; one wrong label photo costs 10 points at once

Structure

56 yes/no checks, each of one kind, weighted by kind: each chair's version, order and parts 25%, the damage and its cause 15%, a product notice 15%, the label photos she sends 15%, the gift's privacy 5%, house style 16% and the basics 9%.

Weights

Each kind's weight is split evenly over its checks, none above 2.5 points. House style is four groups of 4 points, on how items, parts and orders are named and dated in each file.

One error, one charge

Dependent checks are graded on what follows from the attempt's own reading: taking the wrong label photo costs its four label checks, 10 points, not the decisions after it. A version or order naming both options counts as no answer.

Who decides

Scripts decide 46 checks; two blind LLM readers, Opus 5.5 subagents, read 8, and 2 combine a script and a reader item. The messages she was sent are graded with the reply.

Reader agreement

Each of three batches carried three reference answers of known verdict, which both readers matched every time. They split on 2 of 260 readings across the 26 attempts, both ruled to a pass by the task's coordinator on lines set beforehand.

Hand checks

Every lost check was read by hand against the records and the photos before scoring: four grader false positives passed, raising four attempts by 1.0 to 2.3 points, and four rulings applied to every attempt alike.

Formula

Reward = the weighted share of checks passed, from 0 to 1. A missing file fails all the checks that read it, and a keep‑out check fails with no reply, so an empty attempt scores 0.

Reading the chairs held; a wrong label photo and the house style didn't, and none fully solved it

With only the request, the median attempt scored 82% and the best 95%, and none fully solved it; 14 of 24 took a label photo of another chair as the damaged one's. Told the rules, both models tested scored 99%.

Each dot is one attempt; the line marks the median.

With only the request, each model made three attempts, and the table shows their mean. Told the rules, two models made one attempt each.

Told the rules: the request plus one paragraph naming every rule, practice and house habit the checks score and what to ask the customer, never an answer. Each attempt lost one check.

With only the request, the spread between a model's three attempts ran from 0.3 points (Sonnet 5.5) to 26 (Opus 5, whose third attempt lost 13.8 points to one unquoted comma in its decision sheet).

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. The wrong chair's label. 14 of 24 attempts took a label photo of another of her chairs as the damaged chair's, losing 10 points each. 7 questioned it or asked again: all three of Opus 5.5's, two of GPT‑6.1 Sol's, and one each of Opus 5's and GPT‑6 Astra's.

2. Noticed, not asked. Fable 5.1 noted in each of its three attempts that the photo didn't quite fit the damage, and still never asked again. Sonnet 5.5 asked her once in each and took the photo without comment.

3. House style. The four habits cost every attempt 3 to 12 of their 16 points: 23 of 24 dated a timeline row differently from the closed cases, and 19 named her chairs in the note differently. The GPT models lost the most.

4. Practices read, not used. 14 of 24 attempts didn't record one chair's damage the way two closed cases do, among them all three of Opus 5.5's, each made after reading both cases. Four that applied the notice left one of its steps as work still to do.

5. No decision. GPT‑6 Luna opened a single photo across its three attempts, decided no chair and scored 26 to 30%; two of its attempts never messaged her.

6. Hardware that wasn't there. One GPT‑6 Sol attempt saw hardware its photos don't show, placed that chair on the wrong order and promised a free part, losing 15.9 points on that chair and one more check for the unsupported claim.

What an attempt costs

Length

With only the request, the median attempt took 6 minutes and 107 tool calls (22 to 175 across the 24), and about 4.2 million tokens, cached input included.

Cost

Recorded for 12 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the customer's voice and readers aren't included.

What you license

With a licence

Revision 1 in both versions: the hub, its photos and desk tools, the customer's service and fact cards, the grader and its readers' instruction, the answer key, a reference answer that scores 100, and the grader's report on each of the 26 attempts, as baseline results.

Format

Today it runs on a Mac in each agent's own sandbox; packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model with only the request, and told the rules was tested on two models, once each. Fable 5.1, Sonnet 5.5, Opus 5 and GPT‑6.1 Sol lie within 1.7 points, less than most of their own spreads.
• One challenge carries 10 points: taking the wrong chair's label costs four checks at once, so it alone can reorder models whose other losses are close.
• 10 of the 56 checks rely on LLM readers, Opus 5.5 subagents reading Claude and GPT notes alike; the reference answers in each batch measure their drift, not a possible preference for Claude's writing.
• The customer's code misread some messages. After a fix, 13 attempts were voided and made again; in the 17 made on the final code, 13 of 50 messages were answered other than its rules say, none in a way that could change an outcome.
• Each model ran at its agent's default effort: the Claude models at Claude Code's default, GPT‑6.1 Sol at low and the other GPT models at medium.
• Three naive answers (cover both chairs, refuse both, or trust her message), with every file in form but no message to her and no house style, score 33 to 42%; an empty attempt scores 0 and the reference answer 100.
• Attempts ran on one Mac in each agent's sandbox, not a sealed container, on subscription accounts; the isolation probes passed before every attempt.