Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up
ID shop‑the‑room/task02 · revision 1 · results as of 5 November 2026
For a firm that replaces furniture for insurers, the agent works out from a family's dated photos what stood in their flooded bedroom and study, applies the insurer's rule for what to replace, buys the replacements among look‑alikes, and schedules where the crews set them.
- Checks
- 25, 100 points in four groups
- Tested
- 8 models, 3 attempts each
- Best attempt
- 85% (Opus 5.5)
- Median
- 75%
- Full solves
- 0 of 24
- Median attempt
- 13 min
What stood in the flooded rooms, replaced by the insurer's rule, and set back in place
Visual spatial understanding
Each piece's product, colour and size, and where it stood, show only in 13 room photos and the shops' product photos.
Catching errors
Photos from three dates show the rooms changing: a rug and a loveseat in the earlier photos aren't what stood at the flood.
Stateful tool use
Each order charges the card at once with one delivery date, and the fee comes back only if the whole order is cancelled.
Planning ahead
Every delivery must land in a fixed window at the lowest cost, and five pieces arrive in time from only one of two shops.
13 photos, three furniture shops and the firm's card
The agent's world
A text browser into three local furniture shops, a file tool for the schedule and the agent's image viewer; no shell or web.
The shops
148 products in 546 variants, discontinued ones still listed, and a third shop sells some of the brands' pieces under the same names.
Limits
One session of up to 150 minutes and 1,000 turns, with nobody to ask; no attempt reached a limit.
The reward
100 points from code alone: the purchase 59, delivery window and cost 20, positions 16, the schedule 5.
Most attempts scored 70–85%, and two stopped with nothing ordered
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Opus 5.5Anthropic | 82% |
| GPT‑6 AstraOpenAI | 81% |
| Sonnet 5.5Anthropic | 81% |
| GPT‑6.1 SolOpenAI | 77% |
| Opus 5Anthropic | 74% |
| GPT‑6 SolOpenAI | 67% |
| Fable 5.1Anthropic | 64% |
| GPT‑6 LunaOpenAI | 19% |
Given the family's order confirmations and the pack‑out room sheets, both models tested scored 100%.
They buy one step off, take a near copy for the original, and misplace pieces
Visual spatial understanding
The bench was wrong in 96% of attempts and the loveseat in 75%, mostly one step off in colour or size, not the wrong product.
Of the 22 attempts that ordered, 20 put the desk and its chair out of place, and 11 put the loveseat on the wrong wall.
Catching errors
Half of all attempts took one piece for a discontinued near copy and bought a substitute it didn't need, having opened only the near copy's photo sheet.
What an attempt takes, and what the environment includes
What's in it
The shops and their data, the room photos and product sheets, both versions, the full grader, and a reference solution that scores 100.
Format
It runs today on a Mac in each agent's own sandbox, and is packaged as a Harbor task on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials