Furniture warranty claim from a simulated customer's photos
ID letter‑and‑record/task07 · revision 1 · results as of 12 November 2026
At a furniture retailer's customer care desk, a customer writes that two of her dining chairs have broken and sends six phone photos. The agent decides each chair from the photos and the records, asks her for photos only she can take, and writes the reply and records.
- Checks
- 56, weighted by kind
- Tested
- 8 models, 3 attempts each
- Best attempt
- 95% (Opus 5)
- Median
- 82%
- Full solves
- 0 of 24
- Median attempt
- 6 min
Two broken chairs, six phone photos, and labels only she can photograph
Multimodal records
Each chair's version, order and damage exist only in photos, which the agent downloads and views with its own image tool.
Clarifying with people
Only she can photograph the labels under the seats, so the agent must name each chair it needs; one label has come off.
Catching errors
Her belief about which chairs were gifts isn't what the photos show, and one label photo shows a different chair.
Disclosure and honesty
A product notice stays internal: the reply confirms the cover without naming it, and no recall or claim count reaches her.
Unstated requirements
How the team handles gifts and out‑of‑warranty parts, and four writing habits, show only as patterns in 12 closed cases.
A service hub, her photos and a customer who answers 20 messages
The agent's world
A text browser into the company's service hub, through the model's usual coding agent; photos download for its own image tool.
The customer
She answers from 36 fixed cards, worded by an AI model, and sends label photos on request; she stops after 20 messages.
Limits
One session of up to 150 minutes and 1,000 turns; no attempt reached a limit, and nothing changes but her messages.
The reward
56 checks weighted by kind: each chair's version, order and parts 25%, the label photos she sends 15%, house style 16%.
Most attempts scored 75–95%, and three fell below 31%
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Opus 5.5Anthropic | 91% |
| Fable 5.1Anthropic | 84% |
| Sonnet 5.5Anthropic | 83% |
| Opus 5Anthropic | 82% |
| GPT‑6.1 SolOpenAI | 82% |
| GPT‑6 AstraOpenAI | 80% |
| GPT‑6 SolOpenAI | 71% |
| GPT‑6 LunaOpenAI | 28% |
With every rule, practice and house habit spelled out, both models tested scored 99%.
They take the wrong chair's label, slip on the house style, and read practices without using them
Catching errors
58% of attempts took a label photo of another of her chairs as the damaged chair's, losing 10 points each.
Clarifying with people
Fable 5.1 noted in all three attempts that the photo didn't fit the damage, yet never asked her again.
Unstated requirements
House style cost every attempt 3 to 12 of its 16 points; 96% left the order's date off a timeline row that names the order, though every closed case dates such rows.
58% of attempts didn't record one chair's damage the way two closed cases do, all three of Opus 5.5's among them.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 4.2 million tokens, cached input included.
What's in it
The hub with its photos, the customer's service and fact cards, both versions, the full grader, and a reference answer that scores 100.
Format
It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials