AsapiThe AI Agent Trials
Asapi / Catalogue / Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up
Environment datasheet · Insurance replacement

Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up

ID shop‑the‑room/task02 · revision 1 · results as of 5 November 2026

For a firm that replaces furniture for insurers, the agent works out from a family's dated photos what stood in their flooded bedroom and study, applies the insurer's rule for what to replace, buys the replacements among look‑alikes, and schedules where the crews set them.

Checks
25, 100 points in four groups
Tested
8 models, 3 attempts each
Best attempt
85% (Opus 5.5)
Median
75%
Full solves
0 of 24
Median attempt
13 min
What it tests

What stood in the flooded rooms, replaced by the insurer's rule, and set back in place

Visual spatial understanding

Each piece's product, colour and size, and where it stood, show only in 13 room photos and the shops' product photos.

Catching errors

Photos from three dates show the rooms changing: a rug and a loveseat in the earlier photos aren't what stood at the flood.

Stateful tool use

Each order charges the card at once with one delivery date, and the fee comes back only if the whole order is cancelled.

Planning ahead

Every delivery must land in a fixed window at the lowest cost, and five pieces arrive in time from only one of two shops.

How it works

13 photos, three furniture shops and the firm's card

The agent's world

A text browser into three local furniture shops, a file tool for the schedule and the agent's image viewer; no shell or web.

The shops

148 products in 546 variants, discontinued ones still listed, and a third shop sells some of the brands' pieces under the same names.

Limits

One session of up to 150 minutes and 1,000 turns, with nobody to ask; no attempt reached a limit.

The reward

100 points from code alone: the purchase 59, delivery window and cost 20, positions 16, the schedule 5.

Results

Most attempts scored 70–85%, and two stopped with nothing ordered

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each75%0 of 24
ModelMean of 3 attempts
Opus 5.5Anthropic82%
GPT‑6 AstraOpenAI81%
Sonnet 5.5Anthropic81%
GPT‑6.1 SolOpenAI77%
Opus 5Anthropic74%
GPT‑6 SolOpenAI67%
Fable 5.1Anthropic64%
GPT‑6 LunaOpenAI19%

Given the family's order confirmations and the pack‑out room sheets, both models tested scored 100%.

What models get wrong

They buy one step off, take a near copy for the original, and misplace pieces

Visual spatial understanding

  1. The bench was wrong in 96% of attempts and the loveseat in 75%, mostly one step off in colour or size, not the wrong product.

  2. Of the 22 attempts that ordered, 20 put the desk and its chair out of place, and 11 put the loveseat on the wrong wall.

Catching errors

  1. Half of all attempts took one piece for a discontinued near copy and bought a substitute it didn't need, having opened only the near copy's photo sheet.

Running it

What an attempt takes, and what the environment includes

What's in it

The shops and their data, the room photos and product sheets, both versions, the full grader, and a reference solution that scores 100.

Format

It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.