Smart‑home retrofit: purchase plan, wiring and local control software
ID laycode/task10 · revision 2 · results as of 2 November 2026
An old house's owner wants it run from two iPads and a phone, with no single brand's hub or cloud. The agent buys within budget, writes the wiring table and manuals, and leaves software running, so that the owner only plugs in, switches on and pastes codes.
- Checks
- 121, in three weighted parts
- Tested
- 7 models, one attempt each
- Best
- 92% (Opus 5.5)
- Median
- 76%
- Full solves
- 0 of 7
- Median attempt
- 32 min, 99 tool calls
A house run from two iPads and a phone, bought, wired and programmed within budget
Unstated requirements
Only 38 of 121 checks tie to the owner's message; the rest come from house rules, product sheets and basics any system has.
Planning ahead
The purchase plan must fit the budget and still buy a drive that holds 90 days of camera footage.
Stateful tool use
The software runs the devices over their real protocols and has to show their true state, even when one goes offline.
When to act or escalate
The house must shut the water on a leak and the windows in steady rain, yet ignore rain glitches and keep the owner's override.
Disclosure and honesty
The app may claim only what it sees: commands to offline devices aren't shown as done, and visitors see no video.
Agentic coding
121 checks on software left running, security and privacy among them, most judged from the house's own records, not the app's screens.
55 files, a simulated house on a second machine, and an operator in the owner's place
The agent's world
A fresh container with the owner's 55 files, worked through the model's usual coding agent; the software is left on an always‑on Mac mini.
The house
A simulated house on a second machine: cameras, rain and leak sensors, window motors, a water valve and mains circuits, with an append‑only event log.
Grading
An operator model uses the app as owner, visitor and stranger, while the clock jumps for routines, a power cut and footage retention.
The reward
121 checks: plan, wiring and features 60%, edge cases 20%, security and privacy 20%; a device counts only when the house's records show it.
Five attempts scored 59–92%, and two left no app the owner could open
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 92% |
| GPT‑6.1 SolOpenAI | 81% |
| Fable 5.1Anthropic | 78% |
| GPT‑6 AstraOpenAI | 76% |
| Sonnet 5.5Anthropic | 59% |
| GPT‑6 LunaOpenAI | 0% |
| Sonnet 5Anthropic | 0% |
With an IT specialist's 70 requirements added to the request, five of the seven scored 84–96%, and two scored 0.
Visitors keep access, offline cameras look fine, and routines slip after a power cut
Agentic coding
None of the five working apps could make a visitor's access expire.
Two attempts left no app the owner could open: one stopped in under two minutes, and the other never started its package.
Disclosure and honesty
None of the five showed a cut camera as offline in time; one kept showing it as recording, and one had no camera status at all.
When to act or escalate
Only one of the five ran a weekly routine on time, once; two ran a routine missed during a power cut late, and three had no weekly routine the owner could make.
Unstated requirements
Three of the five asked the owner for setup steps outside the allowed list, such as adding the app to the Home Screen.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $7.30 at list prices for the agent model; the operator's and judges' calls are extra.
What's in it
The owner's 55 files, the simulated house and its devices, the circuit simulation, the task in three versions, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the agent's container and the simulated house run on two machines.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials