Flat‑pack assembly manuals with 3D views from unlabelled photos

Ten manuals and their 3D models, worked out from photos alone

Role

Sole builder for a home user, working only from the photos on the user's laptop.

Request

A single app holding a manual per piece: a contents list first, then numbered steps, with a 3D model at each step that the user can turn and zoom.

Deliverable

One offline web app, opened by double‑click with no server or install, with a picker for the 10 pieces.

What the reward depends on

Every panel and hardware group counted right, every join found, hidden parts placed, steps small enough to follow one thing at a time, safety steps, and 3D models that change by step and match the finished piece.

Where the requirements are

The facts (panels, fitting counts, joins, hidden parts, wall kits, the finished shape) are only in 177 photos, 14 to 25 per piece, with no labels or printed text. The short request asks for small steps and a turnable, zoomable 3D view; the rest is ordinary practice.

177 unlabelled photos and nothing else

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI). Internet is on for packages; web search and fetch are off.

Photos

177 read‑only images, rendered from 3D models of the makers' designs to look like phone photos: laid‑out panels, their reverse faces, hardware groups and the shop's photo of each built piece.

Grading

The app opens from a new path in Chromium, with requests logged. An operator model, Opus 5.5, walks every step in a fresh session per piece, more for long manuals, given the shop photo and the answer key's parts and hardware groups, and maps the app's names.

Limits

One build session of 120 minutes, 150 model responses and $20 at list prices.

173 checks: the manuals worth 60 points, the app 40

Structure

Per piece, the manual's checks cover the parts, hardware, joins, build order, hidden parts, steps, wall fixing and a second‑person warning: 60 points in all.

The app

40 points: per piece, the 3D view at each step, highlights, final shape, naming, small steps, orientation, turning, zooming, and Next and Back; app‑wide, opening, the piece picker and loading only its own or public static files.

Who decides

Scripts decide most checks from the operator's records, the saved 3D scene and the request log. Two LLM judges decide highlights and the finished shape and check unsure name matches: GPT‑6.1 Sol and GPT‑5.6 Luna for most, Sonnet 5.5 and Opus 5.5 for 553 re‑asked after a usage limit.

Gates

An app whose first screen doesn't open from file:// fails all 173 checks; a piece that doesn't open fails all 17 of its own.

Formula

Reward = points earned / points of the checks not dropped (100, or 98.5 in the one attempt with a dropped check), with partial credit on the parts, hardware and joins for the share found.

No attempt reached 65% in any version, and none fully solved it

With only the request, scores ran from 21% to 63%. With a hint the mean rose from 50% to 53%; told the rules, to 55%. Every attempt got the hardware counts wrong on all ten pieces.

Each dot is one attempt; the line marks the median.

One attempt per model and version, none made again.

With a hint: the request plus 9 points from a neighbour who assembles flat‑pack furniture for a living.

Told the rules: the request plus 22 product requirements from an IT specialist.

What the models got wrong

Counted over the 7 models, one attempt each, with only the request: 70 manuals.

1. Hardware miscounted. All 70 manuals had wrong hardware‑group counts, taken from the agent's own steps or a template, estimated, or left blank.

2. Joins missed. Every scored piece fell short of the joins pass line, and the strongest attempt found under a third of the real joins.

3. Panels miscounted. 65 of 70 manuals had wrong panel counts; no attempt got more than 2 of its 10 pieces right.

4. Hidden parts and build order. Because so few real joins were found, 65 of 70 manuals left the hidden parts unplaced and 63 of 70 failed the build‑order check, which counts a link only when both of its joins are found in a buildable order.

5. One cabinet for all. 4 of 7 attempts built most pieces from one shared cabinet template.

What an attempt costs

Length

With only the request, the median attempt took 33 minutes and 101 tool calls (5 to 158 across the 7).

Cost

The median attempt cost $7.75 at list prices, from $0.55 to $16.45 across the 7; all 21 attempts came to $168. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the operator's and judges' calls aren't included.

What you license

With a licence

Revision 1 in all three versions: the 177 photos, the answer key's parts, hardware and joins, the operator's prompts, the grader and its LLM judges, a reference app graded in part, and the grader's report on each of the 21 attempts, as baseline results.

Format

The agent's container, and Chromium for grading. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The photos, answer key and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version.
• The reference app is graded only in part: its self‑test covers the app‑wide opening and picker checks, one piece and the files‑loaded check, 20 of the 173 checks.
• The operator is given each piece's parts and hardware groups with their true counts, so the app's names are mapped with help. It is never told the joins, the build order, which parts are hidden, the step count or the safety items.
• Two of the tested models, Sonnet 5.5 and Opus 5.5, judged some questions, including some on their own apps; a step, join or name match is rejected only when both judges fail it.
• One join check in Opus 5.5's attempt with only the request was dropped by the task's maintainer after a grader‑caused miss, so that attempt is scored on 172 checks.