AsapiThe AI Agent Trials
Asapi / Catalogue / Row‑house rooftop solar designed and quoted from plans, imagery and photos
Environment datasheet · Home solar

Row‑house rooftop solar designed and quoted from plans, imagery and photos

ID tb‑tasks/larkrow‑rooftop · revision 1 · results as of 12 November 2026

As a solar installer's desk surveyor with no site visit, the agent identifies the family's house among look‑alikes, measures the neighbours that shade it, finds the tariff that applies, and designs panels, frames, inverters and appliance timers within the family's two limits, then quotes figures the design must earn.

Checks
15, for half the reward; ten‑year gain is the other half
Tested
6 models, one attempt each
Best
99% (Fable 5.2)
Median
71%
Full solves
0 of 6
Median attempt
37 min
What it tests

A fixed‑price solar quote for one house in a street of look‑alikes, with no site visit

Engineering design

The right house's measurements come from plans, images and photos, and the quote may state only what the design earns.

Visual spatial understanding

The roof and the neighbours that shade it are modelled from overhead images, phone photos and plans, with heights read from shadows.

Planning ahead

Half the reward is the family's ten‑year gain, which a neighbour's planned extension and the family's appliance timers both move.

Disclosure and honesty

An honest quote is a hard gate: the quoted ten‑year saving must be within 8% of the grader's figure for the design.

Multimodal records

Five made‑up websites hold the plot register, dated overhead captures, tariffs, a subsidy and weather, beside the family's bills and photos.

Catching errors

The family's postal number, the register and the photos' location point to different plots, and the plan on file isn't what was built.

How it works

28 files, five websites, and no CAD, GIS or simulator

The agent's world

A shell in a Linux container with Python and image tools and five websites served inside it; no CAD, GIS or simulator.

The street

Neighbours shade the small roof, their heights read from shadows, plans and photos; the records disagree on which house it is.

Limits

One session of 300 minutes, against an expert's estimate of eight hours; attempts used 4 to 98 minutes.

The reward

Half is the ten‑year gain in the true street on unseen weather, against the best design; half is 15 checks, six of them gates.

Results

Three designs scored 95–99%, and the other three 47% or less

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each71%0 of 6
ModelReward
Fable 5.2Anthropic99%
Opus 5.5Anthropic99%
Sonnet 5.5Anthropic95%
GPT‑6.1 SolOpenAI47%
GPT‑6 LunaOpenAI37%
GPT‑6 AstraOpenAI3%

Fable 5.2 and Opus 5.5 also passed the task's own bar, which asks for a net gain close to the best design's.

What models get wrong

They pick the wrong house, measure an edge wrong, and keep an outdated height

Catching errors

  1. One attempt took a plot in another block that matched the family's old house number, modelled no shade, and overstated the saving by more than a quarter.

Visual spatial understanding

  1. One measured the stair head's outline from the wrong wall line, 15 cm out, then placed panels at the exact edge of its clearance, failing the placement gate.

  2. Two measured a neighbour's added floor in the overhead images, then kept the older plan height in their model.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.04 to $18.40 for the agent model; grading is a script, with no model calls.

What's in it

The family's files, the installer's documents, the five websites, the grader with its simulator, true scene and unseen weather, and the best design found.

Format

Packaged as a Harbor task; our attempts ran in Harbor, in a Linux container, with the websites served inside it.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.