Flood‑claim replacement from dated photos: what stood, what stays, look‑alikes and set‑up

The guideline is written down; what stood, and where, exists only as pixels

Role

The assistant at a firm that replaces contents for insurers, with the firm's trade accounts at three furniture web shops and its card on file.

Request

Replace a family's water‑damaged furniture, rugs and lamps in a bedroom and a study under the insurer's guideline. Order everything on the firm's card without waiting for anyone, then hand in the replacement schedule.

Deliverable

The orders in the shops, with every live line, its delivery date and the total charged, and a schedule as JSON: one entry per piece at the loss, with the decision, the original, the replacement and its order, and where the piece stands and faces.

What the reward depends on

The exact variant of each piece replaced like for like; the two discontinued originals named, with the one substitute the guideline leaves for each; nothing bought that the guideline keeps; every delivery in the window at the lowest cost; each piece placed and facing where it stood.

Where the requirements are

The guideline, a sketch of both rooms with their sizes and openings, and the floor schedule are in the brief. Which product, colour and size each piece is, and where it stood, show only in 13 photos from three dates and the shops' product photos. Nobody can be asked.

13 dated photos, three web shops and 501 product photo sheets

Interface

19 tools, among them a text browser (8 tools), a file tool (5) that writes only the schedule and notes and checks the schedule's format, and the agent's image viewer. No shell, web or sub‑agents. Claude Code or Codex CLI on a Mac, effort high.

Photos

13 photos of two rendered rooms from three dates in one year, up to 1,568 pixels on the long side with metadata removed, and 501 product photo sheets, saved when the agent opens them.

Shops

Three furniture shops on one local site, two brand stores and one selling some of their pieces under the same names: 148 products in 546 variants, discontinued ones still listed, with search and filters, carts and cancellations.

Checkout

Placing an order charges the card at once, with one delivery fee and one delivery date per order, chosen at checkout from the dates its lines allow. The fee is refunded only when the whole order is cancelled.

Clock

None: a fixed date, and a fixed earliest delivery date for each product at each shop.

Isolation

Each attempt in a fresh work folder. The sandbox refuses the answer key, the grader, the scene and the shops' data files, probed before every attempt, and the shops strip metadata from every photo they serve.

Limits

One session of up to 150 minutes and 1,000 turns, with no spend cap; no attempt reached a limit.

100 points by code: purchase, window and cost, positions and the schedule

Structure

25 checks, 19 yes/no and six on a scale, worth purchase 59, window and cost 20, positions 16 and the schedule 5.

Purchase

Each of six like‑for‑like kinds earns 5.5 for exactly the right variants. Each discontinued piece earns 5 for its original named and 4 for its one substitute. Scope, 8, loses 2 for each kept kind or other line bought.

Window and cost

1.25 for each kind to replace with every line inside the window; 6 for paying no more than the cheapest way to buy the attempt's own lines, one order a shop; 4 for a schedule total equal to the charge. Price and total scale with the kinds right.

Positions

13.5 points share out by the pieces placed close to where they stood and 2.5 by fronts facing the right way. The schedule's 5 points, for its format and its match with the shops' records, count only once it lists most pieces.

Who decides

Code only, from the shops' final state and the schedule; no model judges. The schedule is read leniently except for its own format points. An independent second reading of every lost point reproduced every stored grade and changed no score.

Formula

Reward = points / 100. Two sizes or colours of one kind fail it, and an extra entry of a type counts against its pieces. Ordering nothing and handing in nothing scores 0.

The rules and the money held; the exact pieces and their places didn't, and none fully solved it

With only the request, the median attempt scored 75% and the best 85%, and every attempt lost points both on look‑alikes and on where pieces stood. Given the family's order confirmations and the pack‑out room sheets, both models tested scored 100%.

Each dot is one attempt; the line marks the median.

With only the request, each model made three attempts, and the table shows their mean. Told the rules, two models made one attempt each.

Told the rules: the family's message adds their order confirmations (every piece bought, by brand, model, colour and size, earlier pieces among them) and the pack‑out firm's room sheets (each piece's room, centre and facing). Both attempts lost nothing.

With only the request, the spread between a model's three attempts ran from 3 points (Sonnet 5.5) to 20 (Fable 5.1), and 57 for GPT‑6 Luna, which stopped twice with nothing ordered.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. One step off. The bench was wrong in 23 of 24 attempts and the loveseat in 18, mostly by one step of colour or size rather than the wrong product. The six like‑for‑like kinds took 61% of all points lost.

2. A discontinued near copy. 12 of the 22 that ordered took one piece for a discontinued near copy and bought a substitute it didn't need. Every attempt that did so had opened the near copy's photo sheet and not the true piece's.

3. Pieces out of place. Every attempt lost points on positions; those that ordered placed 3 to 12 of the 15 pieces close enough to count. The desk and its chair were out in 20 of 22, and 11 put the loveseat on the wrong wall.

4. Kept or replaced wrongly. Two attempts bought a piece the guideline keeps, and two left one and three damaged pieces unreplaced, taking pieces with a base to the floor for look‑alikes on legs or finding no substitute.

5. Stopping at once. 2 of 24 attempts, both GPT‑6 Luna's, stopped within two minutes with nothing ordered and no schedule; one said its workspace was read‑only, without trying the file tool that worked in its finished attempt.

What an attempt costs

Length

With only the request, the median attempt took 13 minutes and about 190 tool calls (15 to 367 across the 24).

Cost

Recorded for 12 of the 24 attempts with only the request, the four Anthropic models', at list prices by Claude Code's own count; the OpenAI models' agent logs tokens but not cost. With half unrecorded, we give no median or range.

What you license

With a licence

Revision 1 in both versions: the shops and their data, the room photos and product sheets, the grader, the answer key, a reference solution that scores 100, and the grader's report on each of the 26 attempts, as baseline results.

Format

Today it runs on a Mac in each agent's own sandbox; packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Ordering nothing scores 0, and eight scripted naive buyers, from the first search hit or the cheapest product per kind to scope decided by category and a layout from the sketch, score 1 to 9%.
• Some sizes can be right by a default choice rather than a reading, so a right size proves less than it seems; such a default can only raise a score.
• Each product's photo sheets show it at its first listed size, and no page says so; on one rug this misled one attempt, which lost 5.5 points.
• An independent second reading found 49 losses where a natural reading of the photos gives the attempt's answer; crediting them all would lift the top four models by 7 to 11 points. It is reported, not scored.
• Told the rules was tested on two models, once each.
• The rooms were rendered from our own 3D scenes, so their answers are exact, and their photos cleaner than a real home's.
• Delivery dates are fixed per product and shop, so the job tests choosing shops, orders and dates, not reacting to stock or delays.