Smart‑home retrofit: purchase plan, wiring and local control software

The owner can only plug in, switch on and paste codes; the agent does the rest

Role

Sole builder for a homeowner, with an always‑on Mac mini for the software and an electrician who wires only from the agent's table.

Request

Control the house from an iPad at home and a phone when out, with no lock‑in to one brand, hub or cloud, keep local control when the internet is down, and do all the setup, pairing, testing and recovery the owner can't.

Deliverable

A priced purchase plan, a wiring table, a wiring manual with the mains steps marked for an electrician, a plain user manual, and the software left running on the Mac mini.

What the reward depends on

A plan within budget whose wiring works in simulation; everyday control, scenes, routines and a visitor view; cameras whose recordings are kept and played back for the period the owner asks; power and run‑time limits; safe recovery after faults and power cuts; security.

Where the requirements are

In the owner's 55 files: the message, a world manual with the house rules, one product sheet per product, an electrician's survey, a format sheet and the catalogue, plus what a professional builds unasked. The house's own timing rules show only in the world manual.

A simulated house where only bought units exist and power follows the wiring

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI). Internet is on for packages; web search and fetch are off.

The house

A practice house on a second machine: simulated smart‑home devices, an infrared air‑conditioner bridge, two cameras, a rain sensor, two window motors, a leak sensor and water valve, an energy meter, mains circuits, a router and an append‑only event log out of the agent's reach.

Files

55, read‑only: 53 text files, a catalogue and seven pages of appliance and switch photos that no check depends on.

Grading

An operator model uses the app as the owner on an iPad‑sized browser and an outside phone, as a visitor and as a stranger, limited in setup to what the owner may do. The clock jumps forward for routines, a power cut and the footage‑retention test.

Limits

One build session of 120 minutes, 150 model responses and $20 at list prices.

121 checks in three weighted sections, mostly decided from the simulator's records

Structure

Build: 61 checks share 60 points (the plan, wiring and manuals; everyday control; automations, cameras and power). Edge cases: 40 at half a point, from faults to interruptions. Security and privacy: 20 at a point each.

Who decides

Scripts decide 111 checks from the simulator's records: the catalogue, a circuit simulation of the wiring, device states, the event log, network captures and probes, the camera files and the operator's saved pages. Two LLM judges decide 10: the setup steps, eight manual checks and the air‑conditioner's screen label.

What counts

A device, recording or alert counts only when the simulator's records show it; the app's screen or manual is the claim tested, never the proof. Fresh copies of the house start from the agent's output, and the grader changes timed house rules only through the owner's settings in the app.

Formula

Reward = (60 × build passed / 61 + 0.5 × edge passed + security passed) / 100. An app the owner can't open fails all 121 checks, the plan and wiring included.

Five working apps, two attempts with none; no attempt fully solved it

With only the request, the five working apps scored 59% to 92%, and two attempts left no app the owner could open. With a hint the mean went from 55% to 57%, though the median fell from 76% to 68%; told the rules, the mean was 63%.

Each dot is one attempt; the line marks the median.

One graded attempt per model and version; two Claude attempts stopped by subscription limits (Fable 5.1 with a hint, Opus 5.5 told the rules) were made again from the start.

With a hint: the request plus points from a friend who installs smart homes (10 in the hint file in the package; the task's README says 6).

Told the rules: the request plus 70 requirements from an IT specialist.

What the models got wrong

Counted over the 5 apps the owner could open, with only the request; the other 2 attempts left no app and fail all their checks.

1. Visitor access that never ends. All 5 gave visitors access with no way to make it expire.

2. Routines after a power cut. 2 of 5 ran a routine missed during a power cut late, and 3 had no weekly routine the owner could make; only 1 of 5 ran a weekly routine at its time, once.

3. Offline cameras not shown. All 5 missed showing a cut camera as offline in time: one kept showing it as recording, and one had no camera status at all.

4. More than the owner can do. 3 of 5 asked the owner for setup steps outside the allowed list, such as adding the app to the Home Screen or switching the phone's tunnel on.

5. No app at all. The other 2 attempts left none the owner could open: one stopped in under two minutes, the other never started its package.

What an attempt costs

Length

With only the request, the median attempt took 32 minutes and 99 tool calls (19 to 126 across the 7).

Cost

The median attempt cost $7.34 at list prices, from $1.38 to $17.53 across the 7; all 21 attempts came to $159. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the operator's and judges' calls aren't included.

What you license

With a licence

Revision 2 in all three versions: the owner's files, the simulated house and its devices, the circuit simulation, the operator's prompts, the grader and its LLM judges, and the grader's report on each of the 21 attempts, as baseline results. No reference app is stored.

Format

The agent's container and the simulated house on a second machine. Packaged as a Harbor task on request.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, house and grader aren't published.

What we haven't measured, and what's uneven

• One graded attempt per model and version, so we haven't measured how much attempts vary here.
• No reference app is stored, and the task's README says a full scored reference grading remains untested.
• The task's README names Opus 5.5 as the operator and Sonnet 5.5 and GPT‑5.6 Luna as the judges; from 30 October the grader could hand the operator's work and the judging to one Codex session, and the repo doesn't say which final verdicts came from which.
• The hint file in the package lists 10 points while the task's README says 6; which text the attempts saw can't be checked from the repo.