DIY smart‑lighting plan from notes, photos and one shop's catalogue
ID workaround/task04 · revision 2 · results as of 18 November 2026
A handy homeowner will fit six smart ceiling lights in his bedroom. From his message, notes, nine photos and one shop's catalogue, the agent plans the layout, the controls, the wiring and a priced shopping list, and hands in one offline guide he can follow up a ladder.
- Checks
- 26, 1 point each; a parts table the scripts can't read fails all 14 scripted checks
- Tested
- 8 models, one attempt each
- Best
- 94% (Fable 5.2 and GPT‑6 Astra)
- Median
- 89%
- Full solves
- 0 of 8
- Median attempt
- 7 min, 24 tool calls
Six smart ceiling lights, and a guide he can follow up a ladder
Engineering design
Six lights must suit his wiring and switches, clear the ceiling fan, miss the joists and keep their beams out of a reader's eyes.
Disclosure and honesty
The guide for a homeowner up a ladder may make no false technical claim, and state nothing the catalogue, notes or chat contradict.
Unstated requirements
His message gives wishes, not limits: beams must miss the reader's eyes and the fan's blades, and light the desk from in front.
Notes, nine photos, a 35‑item catalogue and a homeowner to message
The agent's world
A shell in a container with his notes, nine photos and one shop's catalogue, and one command to message him and hand in.
The homeowner
An AI model plays him from a brief of what he wrote and saw; he invents nothing, volunteers nothing and suggests no design.
Limits
One session of up to 120 minutes, 150 turns and $20 at list prices; every attempt handed in a guide.
The reward
26 checks: scripts score 14 from the guide's parts table, which must be readable, and two AI judges 12, a split paying half.
Six of eight attempts scored 88–94%, and one handed in after three minutes at 31%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.2Anthropic | 94% |
| GPT‑6 AstraOpenAI | 94% |
| GPT‑6.1 SolOpenAI | 90% |
| Sonnet 5.5Anthropic | 90% |
| Fable 5.1Anthropic | 88% |
| Opus 5.5Anthropic | 88% |
| Sonnet 5Anthropic | 83% |
| GPT‑6 LunaOpenAI | 31% |
With an electrician's step‑by‑step method added to the request, the mean rose to 88% and one attempt met all 26 checks.
They make false claims, shine lights in the reader's eyes, and leave joints unspecified
Disclosure and honesty
In every guide, at least one judge found a false technical claim, such as a dust cup that comes only with the other hole saw.
Engineering design
63% failed the check that no beam reaches the reader's eyes, one only because the scripts couldn't read its parts table.
In 75% of guides, at least one judge found a wire joint not taken to a named terminal in the right colours.
What an attempt costs, and what the environment includes
Cost
Attempts cost $0.13 to $7.83 at list prices for the agent model; the homeowner's replies and the judges are extra.
What's in it
The message, notes, photos and catalogue, the homeowner's brief, all three versions, the full grader, and a reference layout.
Format
The agent runs in Docker and the homeowner in a second container; packaged as a Harbor task on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials