Family move‑in from listing photos and papers: rooms, look‑alike purchases and full layouts

Three standards are written down; the rest of a careful move manager's job is in photos and papers

Role

The assistant to a move manager, with the firm's trade accounts at three furniture web shops and its card on file, getting a house ready before a family of six moves in.

Request

Decide who sleeps and works in which of six rooms, buy what the family still needs, decide which of their pieces come in and which go to storage, and lay out every room, within their budget and with every delivery on one of two days.

Deliverable

The orders in the shops and a plan as JSON: each room's use, and one entry per piece, bought or brought from the inventory, plus the pets' two pieces, each with its centre and facing in its room's frame.

What the reward depends on

Each use in its right room; twelve purchases, six of them the same product, colour and size as before; which optional pieces come along; the family's needs in their rooms; layouts that keep the written clearances and the care the house calls for; the budget and the days.

Where the requirements are

Three placement standards are written, and the brief says they aren't the whole job. The rest is spread over the family's papers, 17 listing photos, two plan drawings with room sizes but no other dimensions, and five photos of their old homes. Nobody can be asked.

24 photos of a house and two old homes, the family's papers and three web shops

Interface

19 tools, among them a text browser (8 tools), a file tool (5) that writes only the plan and notes and checks the plan's format, and the agent's image viewer. No shell, web or sub‑agents. Claude Code or Codex CLI on a Mac, effort high.

Photos

17 listing photos of the empty rendered house, two listing‑style plan drawings with doors, swings and windows drawn but not dimensioned and no scale bar, and five photos of the family's old homes, all with metadata removed.

Papers

In the brief: a family group chat, discharge instructions, a night‑shift schedule, a child's allergy plan, two vet records, the listing's text and a measured inventory of the 27 pieces the movers bring.

Shops

Three furniture shops on one local site: 173 products in 648 variants, with search and filters, product pages with a variant table of dimensions, 569 product photo sheets saved when opened, carts and cancellations.

Checkout

Placing an order charges the card at once, with one delivery fee and one delivery date per order, chosen at checkout from the dates its lines allow.

Clock

None: a fixed date, and a fixed earliest delivery date for each product at each shop.

Isolation

Each attempt in a fresh work folder. The sandbox refuses the answer key, the grader, the house and the shops' data files, probed before every attempt.

Limits

One session of up to 150 minutes and 1,000 turns, with no spend cap; no attempt reached a limit.

100 points by code in seven groups; the care beyond the standards is scored in proportion

Structure

45 checks, 34 yes/no and 11 on a scale, worth rooms 25, purchase 15, the family's own pieces 10, the family's needs 12, layout 33, money and dates 3 and the plan's form 2.

Rooms and purchases

5 points for each of five uses in its room. Twelve purchases carry 12.75 points, each passed only by the right product, colour and size where the family wants it as before; scope, 2.25, loses 0.75 for each unit beyond the list.

Pieces and needs

Eleven optional inventory lines share 10 points, each right when kept in its owners' room or rightly left out. The needs, 12 points: a walker's space and route by one bed, a small child's windows, the office window, and the pets' places.

Layout

33 points as shares over the five rooms: valid pieces 8, beds' clear space 6, door swings 5, windows 3, heating registers and a heater 8, the family's pieces in their owners' rooms 3. Care beyond the written standards never makes a piece invalid.

Who decides

Code only, from the shops' final state and the plan; no model judges. After the attempts, a second reading set two checks to the brief's own words and every attempt was graded again; the scores here are that regrade, with the first grades kept beside them.

Formula

Reward = points / 100. Budget and delivery‑day points scale with the needs met. Ordering nothing and writing no plan earns only the optional pieces rightly left out, 5%.

Rooms, money and dates held; the care the house calls for didn't, and none fully solved it

With only the request, the median attempt scored 83% and the best 92%. 19 of 23 plans put every use in its room, but 22 covered a heating register or the heater. Given the needs list, the site rules and a measured plan, both models tested scored 100%.

Each dot is one attempt; the line marks the median.

With only the request, each model made three attempts, and the table shows their mean. Told the rules, two models made one attempt each.

Told the rules: the family's papers are replaced by a needs list (who sleeps where and why, what to buy by product, colour and size, which pieces go in), the move manager's site rules for the house and a measured plan of every room. Both attempts lost nothing.

With only the request, the spread between a model's three attempts ran from 2.7 points (GPT‑6.1 Sol) to 15 (GPT‑6 Sol), and 51 for GPT‑6 Luna, which stopped once after 75 seconds.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Care beyond the standards. Of the 23 plans, 22 put a piece on a heating register or the heater, 21 left something in front of the four‑year‑old's windows and 17 put a piece in a door's swing. This care took 32% of all points lost.

2. Sizes from the shop, not the photos. Of 61 lines bought for the three as‑before products whose size counts, 31 were the wrong size, 22 of them the size the product page says its photos show. The client armchairs were wrong in 21 of the 23 attempts that ordered.

3. Beyond the list. 22 of the 23 attempts that ordered bought 1 to 6 units beyond the family's list, each a wrong size or a look‑alike of a piece wanted as before, losing the scope part.

4. A step only the photos show. 4 of 23 plans, all GPT‑6 Sol's or GPT‑6 Luna's, swapped two rooms' uses, putting someone in a room whose steps rule it out for them; none wrote a word about the steps, which only the listing photos show.

5. Where a door leads. 15 of 23 plans put the dog's bed in a room without the door its need calls for. Opus 5.5 and GPT‑6 Astra placed it right in all six of their attempts; Opus 5, Fable 5.1 and Sonnet 5.5 in none of their nine.

6. Stopping at once. One GPT‑6 Luna attempt stopped after 75 seconds with nothing ordered and no plan, scoring 5%.

What an attempt costs

Length

With only the request, the median attempt took 19 minutes and 157 tool calls (29 to 212 across the 24).

Cost

Recorded for 12 of the 24 attempts with only the request, the four Anthropic models', at list prices by Claude Code's own count; the OpenAI models' agent logs tokens but not cost. With half unrecorded, we give no median or range.

What you license

With a licence

Revision 1 in both versions: the shops and their data, the house's photos and plans, the product sheets, the grader, the answer key, a reference solution that scores 100, and the grader's report on each of the 26 attempts, as baseline results.

Format

Today it runs on a Mac in each agent's own sandbox; packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Ordering nothing scores 5%, and four scripted naive plans, from the room assignment alone to the first search hits piled at each room's centre, score 11 to 22%.
• Careful plans that each miss one source score 67% blind to the family's needs, 81% blind to the photos and 88% keeping only the written standards.
• Only two of the as‑before purchases turn on reading a size from an old photo, so the purchase points test photo reading less than their number suggests.
• Reaching a closet isn't checked, and the key's own plan stands a piece within 24 in of four of the five closets in the planned rooms.
• A second reading after the attempts found two checks stricter than the brief. With both fixed, every attempt was graded again, moving model means by -1.7 to +1.9 points; this page shows the regrade, with the first grades kept beside it.
• Its other rulings found 16 losses where the cue is there but thin; crediting them would lift six models by 0.5 to 1.9 points. They are reported, not scored.
• Told the rules was tested on two models, once each. Its needs list wrongly says only one size of a bedside‑table pair suits the room; it cost neither attempt a point.
• The house was rendered from our own 3D scene, so its answers are exact, and its photos cleaner than a real listing's.