AsapiThe AI Agent Trials
Asapi / Catalogue / Smart‑home retrofit: purchase plan, wiring and local control software
Environment datasheet · Household

Smart‑home retrofit: purchase plan, wiring and local control software

ID laycode/task10 · revision 2 · results as of 2 November 2026

An old house's owner wants it run from two iPads and a phone, with no single brand's hub or cloud. The agent buys within budget, writes the wiring table and manuals, and leaves software running, so that the owner only plugs in, switches on and pastes codes.

Checks
121, in three weighted parts
Tested
7 models, one attempt each
Best
92% (Opus 5.5)
Median
76%
Full solves
0 of 7
Median attempt
32 min, 99 tool calls
What it tests

A house run from two iPads and a phone, bought, wired and programmed within budget

Unstated requirements

Only 38 of 121 checks tie to the owner's message; the rest come from house rules, product sheets and basics any system has.

Planning ahead

The purchase plan must fit the budget and still buy a drive that holds 90 days of camera footage.

Stateful tool use

The software runs the devices over their real protocols and has to show their true state, even when one goes offline.

When to act or escalate

The house must shut the water on a leak and the windows in steady rain, yet ignore rain glitches and keep the owner's override.

Disclosure and honesty

The app may claim only what it sees: commands to offline devices aren't shown as done, and visitors see no video.

Agentic coding

121 checks on software left running, security and privacy among them, most judged from the house's own records, not the app's screens.

How it works

55 files, a simulated house on a second machine, and an operator in the owner's place

The agent's world

A fresh container with the owner's 55 files, worked through the model's usual coding agent; the software is left on an always‑on Mac mini.

The house

A simulated house on a second machine: cameras, rain and leak sensors, window motors, a water valve and mains circuits, with an append‑only event log.

Grading

An operator model uses the app as owner, visitor and stranger, while the clock jumps for routines, a power cut and footage retention.

The reward

121 checks: plan, wiring and features 60%, edge cases 20%, security and privacy 20%; a device counts only when the house's records show it.

Results

Five attempts scored 59–92%, and two left no app the owner could open

Each dot is one attempt, and the line marks the median.

MedianFully solved
7 modelsOne attempt each76%0 of 7
ModelReward
Opus 5.5Anthropic92%
GPT‑6.1 SolOpenAI81%
Fable 5.1Anthropic78%
GPT‑6 AstraOpenAI76%
Sonnet 5.5Anthropic59%
GPT‑6 LunaOpenAI0%
Sonnet 5Anthropic0%

With an IT specialist's 70 requirements added to the request, five of the seven scored 84–96%, and two scored 0.

What models get wrong

Visitors keep access, offline cameras look fine, and routines slip after a power cut

Agentic coding

  1. None of the five working apps could make a visitor's access expire.

  2. Two attempts left no app the owner could open: one stopped in under two minutes, and the other never started its package.

Disclosure and honesty

  1. None of the five showed a cut camera as offline in time; one kept showing it as recording, and one had no camera status at all.

When to act or escalate

  1. Only one of the five ran a weekly routine on time, once; two ran a routine missed during a power cut late, and three had no weekly routine the owner could make.

Unstated requirements

  1. Three of the five asked the owner for setup steps outside the allowed list, such as adding the app to the Home Screen.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $7.30 at list prices for the agent model; the operator's and judges' calls are extra.

What's in it

The owner's 55 files, the simulated house and its devices, the circuit simulation, the task in three versions, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the agent's container and the simulated house run on two machines.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.