AsapiThe AI Agent Trials
Asapi / Catalogue / Furniture warranty claim from a simulated customer's photos
Environment datasheet · Customer service

Furniture warranty claim from a simulated customer's photos

ID letter‑and‑record/task07 · revision 1 · results as of 12 November 2026

At a furniture retailer's customer care desk, a customer writes that two of her dining chairs have broken and sends six phone photos. The agent decides each chair from the photos and the records, asks her for photos only she can take, and writes the reply and records.

Checks
56, weighted by kind
Tested
8 models, 3 attempts each
Best attempt
95% (Opus 5)
Median
82%
Full solves
0 of 24
Median attempt
6 min
What it tests

Two broken chairs, six phone photos, and labels only she can photograph

Multimodal records

Each chair's version, order and damage exist only in photos, which the agent downloads and views with its own image tool.

Clarifying with people

Only she can photograph the labels under the seats, so the agent must name each chair it needs; one label has come off.

Catching errors

Her belief about which chairs were gifts isn't what the photos show, and one label photo shows a different chair.

Disclosure and honesty

A product notice stays internal: the reply confirms the cover without naming it, and no recall or claim count reaches her.

Unstated requirements

How the team handles gifts and out‑of‑warranty parts, and four writing habits, show only as patterns in 12 closed cases.

How it works

A service hub, her photos and a customer who answers 20 messages

The agent's world

A text browser into the company's service hub, through the model's usual coding agent; photos download for its own image tool.

The customer

She answers from 36 fixed cards, worded by an AI model, and sends label photos on request; she stops after 20 messages.

Limits

One session of up to 150 minutes and 1,000 turns; no attempt reached a limit, and nothing changes but her messages.

The reward

56 checks weighted by kind: each chair's version, order and parts 25%, the label photos she sends 15%, house style 16%.

Results

Most attempts scored 75–95%, and three fell below 31%

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each82%0 of 24
ModelMean of 3 attempts
Opus 5.5Anthropic91%
Fable 5.1Anthropic84%
Sonnet 5.5Anthropic83%
Opus 5Anthropic82%
GPT‑6.1 SolOpenAI82%
GPT‑6 AstraOpenAI80%
GPT‑6 SolOpenAI71%
GPT‑6 LunaOpenAI28%

With every rule, practice and house habit spelled out, both models tested scored 99%.

What models get wrong

They take the wrong chair's label, slip on the house style, and read practices without using them

Catching errors

  1. 58% of attempts took a label photo of another of her chairs as the damaged chair's, losing 10 points each.

Clarifying with people

  1. Fable 5.1 noted in all three attempts that the photo didn't fit the damage, yet never asked her again.

Unstated requirements

  1. House style cost every attempt 3 to 12 of its 16 points; 96% left the order's date off a timeline row that names the order, though every closed case dates such rows.

  2. 58% of attempts didn't record one chair's damage the way two closed cases do, all three of Opus 5.5's among them.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 4.2 million tokens, cached input included.

What's in it

The hub with its photos, the customer's service and fact cards, both versions, the full grader, and a reference answer that scores 100.

Format

It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.