DIY smart‑lighting plan from notes, photos and one shop's catalogue

The message gives his wishes; the room, the wiring and the parts set the limits

Role

An online helper to a handy homeowner who will do the wiring himself under an exemption, with an inspector to certify and connect it. The agent works from files and one command, wa.

Request

Six smart ceiling lights in three rows of two, switchable in groups and all off from the bed, warm, lighting the open wardrobe without a shadow on the desk, from one shop within his budget; his partner's habit of switching off at the door mustn't break it.

Deliverable

One offline HTML page for his phone: a ceiling plan with groups, wiring pictures, numbered steps with pictures giving each wire colour to each terminal, and a priced list with full specs. Handing it in ends the attempt.

What the reward depends on

Beams clear of the reader's eyes and the fan blades, lighting the desk and the wardrobe; holes clear of joists in a straight grid; controls that survive the door switch, the old dimmer and weak Wi‑Fi; safe isolation, earths and correct joints; no false claim.

Where the requirements are

In his notes on the switchboard, roof space, joists and switch box, in nine photos, and in the catalogue's specs and limits. He can be messaged, but knows only what he wrote and saw.

His message, notes, photos and one shop's catalogue, with the homeowner played by an LLM

Interface

A shell in a fresh Docker container, with the agent's own CLI and one client, wa, to read his message, message him and hand in. His notes, photos and the catalogue are files in the container.

Isolation

The wa server and the homeowner run in a second container, so his brief and the grader can't be read. The container could reach the internet; web search and fetch were off.

Files

His notes on the switchboard, the roof space and joists, the junction box, the switch box and the old light; nine photos of the room, the view from the pillow, the switches and the roof space.

Catalogue

35 items with codes, prices and specs: 12 lights that differ in beam, colour, insulation rating, cut‑out and control; wiring parts; switches, remotes and a hub, some with a minimum load or a driver limit; and hole saws.

Simulated person

The homeowner, played by Sonnet 5.5, by message only: one fresh call per message, from a brief of what he wrote and saw. He invents nothing, volunteers nothing, suggests no design, and can switch the breaker off or on.

Clock

None: his replies come when the model call returns. No message cap.

Limits

One session of up to 120 minutes, 150 turns and $20 at list prices; every attempt handed in a guide, and none reached a limit.

26 checks of equal weight: 14 by script on the judges' extractions, 12 by the two judges

Structure

26 checks in five families: light placement (7), light choice and shopping list (6), controls and the door switch (8), safe wiring (3), and a guide he can follow that is true (2).

Extraction

Each judge pulls the lights' positions and specs, the shopping list and the switch‑box devices out of the page. Scripts score 14 checks on each extraction: beam cones against the fan, the eyes, the desk and the wardrobe; holes against joists; the catalogue's prices and limits.

Judged checks

Two model judges, Opus 5.5 and GPT‑6.1 Sol, with no tools, mark 12 checks against a fact sheet of the house and the chat. Both are among the models tested, and judged their own guides too.

Agreement

A check passes only when it passes on both judges' readings. Splits, and false‑claims fails from both, are ruled by the environment's author: 28 of the 208 verdicts with only the request.

Lenient reading

Any wording, layout and order. Embedded photos are shortened before the judges read the page, so a fail resting only on them doesn't count.

Formula

Reward = checks passed / 26, each worth the same, with no gates.

Every guide handled the door switch; glare at the pillow and false claims cost most of the points lost

With only the request, scores ran from 69% to 96%, median 92%, and no attempt met all 26 checks; 6 of 8 put a beam in the reader's eyes. With a hint the mean stayed at 89%; told the rules it rose to 98%, and three met all 26.

Each dot is one attempt; the line marks the median.

Each model made one attempt in each version.

With a hint: the request plus a note from an electrician friend with 12 pieces of advice, from isolating and testing dead to checking each light against the reader's eyes, the fan, the desk and the wardrobe.

Told the rules: the request plus an electrician's method of about 2,300 words in steps, restating the house facts and giving the checks' own limits as numbers.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. Glare at the pillow. 6 of 8 put a light whose beam reaches the reader's eyes; with a hint, 7 of 8 did, though the note named the pillow. Told the rules, none did.

2. False claims. 5 of 8 guides stated a part detail or house fact that the catalogue or notes contradict, such as an accessory that comes only with a neighbouring item. Told the rules, it was the only loss for five of eight.

3. Steps without pictures. 3 of 8 guides gave steps with no pictures, one written by a model that never looked at its own page.

4. Beams treated as points. 2 of 8 checked lights against the fan by their centres, not their beams, and sent a wide beam into the blade sweep.

5. No check before hand‑in. GPT‑6 Luna wrote the guide in one pass and handed in after three minutes, at 69%: more LED drivers on one switch than it's rated for, a row of lights on a joist and no hole saw.

What an attempt costs

Length

With only the request, the median attempt took 9 minutes of real time and about 20 tool calls (10 to 83 across the 8).

Cost

Claude Code's own figures at list prices for its 5 attempts with only the request: $0.90 to $6.80. Agent model only: the homeowner's replies and the judges aren't included.

OpenAI's estimates

The runner's estimates for the 3 Codex CLI attempts, their logged tokens at list prices: $0.27 to $1.54. All attempts ran on plan sign‑ins, so none was billed per call.

What you license

With a licence

Revision 1 in all three versions: the message, notes, photos and catalogue, the homeowner's brief and the wa server, the grader with the judges' instructions and fact sheet, a reference layout passing its 14 script checks, and the grader's report on each of the 24 attempts, as baseline results.

Format

The agent runs in a Docker container; the wa server and the homeowner run in a second container, reached through one port. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The files, grader and judges' fact sheet aren't published.

What we haven't measured, and what's uneven

• Each model made one attempt in each version, so a model's score includes the variation between attempts.
• The two judges are among the models tested and judged their own guides; splits, and false‑claims fails from both, were ruled by the environment's author.
• The budget check reads only the priced list: parts named only in the guide's notes or steps aren't counted as purchases, and some attempts passed it only under that rule.
• His notes already hold every fact the homeowner can give, so messaging him adds none; a model plays him, so his wording varies between attempts.
• The judges see embedded photos shortened, so they judge the guide's drawings and text, not its photos.
• The board was first run in a setting where owner wiring isn't legal; the house was moved and every attempt made again. The scores are from the second board.
• The self‑test checks a reference layout on the 14 script checks and the grading logic with stand‑in judges; no reference guide has been graded by the real judges.