Row‑house rooftop solar designed and quoted from plans, imagery and photos
ID tb‑tasks/larkrow‑rooftop · revision 1 · results as of 12 November 2026
As a solar installer's desk surveyor with no site visit, the agent identifies the family's house among look‑alikes, measures the neighbours that shade it, finds the tariff that applies, and designs panels, frames, inverters and appliance timers within the family's two limits, then quotes figures the design must earn.
- Checks
- 15, for half the reward; ten‑year gain is the other half
- Tested
- 6 models, one attempt each
- Best
- 99% (Fable 5.2)
- Median
- 71%
- Full solves
- 0 of 6
- Median attempt
- 37 min
A fixed‑price solar quote for one house in a street of look‑alikes, with no site visit
Engineering design
The right house's measurements come from plans, images and photos, and the quote may state only what the design earns.
Visual spatial understanding
The roof and the neighbours that shade it are modelled from overhead images, phone photos and plans, with heights read from shadows.
Planning ahead
Half the reward is the family's ten‑year gain, which a neighbour's planned extension and the family's appliance timers both move.
Disclosure and honesty
An honest quote is a hard gate: the quoted ten‑year saving must be within 8% of the grader's figure for the design.
Multimodal records
Five made‑up websites hold the plot register, dated overhead captures, tariffs, a subsidy and weather, beside the family's bills and photos.
Catching errors
The family's postal number, the register and the photos' location point to different plots, and the plan on file isn't what was built.
28 files, five websites, and no CAD, GIS or simulator
The agent's world
A shell in a Linux container with Python and image tools and five websites served inside it; no CAD, GIS or simulator.
The street
Neighbours shade the small roof, their heights read from shadows, plans and photos; the records disagree on which house it is.
Limits
One session of 300 minutes, against an expert's estimate of eight hours; attempts used 4 to 98 minutes.
The reward
Half is the ten‑year gain in the true street on unseen weather, against the best design; half is 15 checks, six of them gates.
Three designs scored 95–99%, and the other three 47% or less
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.2Anthropic | 99% |
| Opus 5.5Anthropic | 99% |
| Sonnet 5.5Anthropic | 95% |
| GPT‑6.1 SolOpenAI | 47% |
| GPT‑6 LunaOpenAI | 37% |
| GPT‑6 AstraOpenAI | 3% |
Fable 5.2 and Opus 5.5 also passed the task's own bar, which asks for a net gain close to the best design's.
They pick the wrong house, measure an edge wrong, and keep an outdated height
Catching errors
One attempt took a plot in another block that matched the family's old house number, modelled no shade, and overstated the saving by more than a quarter.
Visual spatial understanding
One measured the stair head's outline from the wrong wall line, 15 cm out, then placed panels at the exact edge of its clearance, failing the placement gate.
Two measured a neighbour's added floor in the overhead images, then kept the older plan height in their model.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.04 to $18.40 for the agent model; grading is a script, with no model calls.
What's in it
The family's files, the installer's documents, the five websites, the grader with its simulator, true scene and unseen weather, and the best design found.
Format
Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the websites served inside it.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials