Row‑house rooftop solar designed and quoted from plans, imagery and photos

Find the house, measure the street, quote what the meter will show

Role

The desk surveyor of a home‑solar installer that quotes from the desk: a fixed price in a day, no site visit, and a saving the meter should then show.

Request

Issue the design and quote for a family of four in a street of near‑identical row houses, who asked by email with their bills, roof photos, a spending ceiling and a payback limit.

Deliverable

A design file (the plot, panel arrays with frames, tilt, facing and position, inverters, an optional battery, timers for the appliances the family let move) and a quote with seven figures.

What the reward depends on

The family's ten‑year net gain in the true street on unseen weather, against the best design the task's search found; their limits and the standard's rules met; and a quote the design actually earns.

Where the requirements are

In the installer's standard, engineering basis, catalogue and price book, the family's thread, and five websites: the city's plan register and imagery portal, the electricity company, the subsidy agency and the weather service.

Five websites, three overhead images, and nobody to ask

Interface

A shell in a Linux container with Python, numpy, Pillow, OpenCV, exiftool, PDF tools and curl. There is no CAD, GIS or photogrammetry tool, and no simulator.

Websites

Five made‑up sites served inside the container and read with curl: plot pages, plans and bye‑laws; three dated overhead captures as image tiles; tariff orders and regulations; a subsidy scheme; half‑hourly weather in UTC.

Network

None beyond those sites. Codex ran with web search off; Claude Code's web tools were on for the three Claude models and could reach nothing.

Files

28 in the workspace: the family's thread, 12 PDF bills, 6 roof photos with camera tags, the installer's 4 documents and 5 site links. The sites hold 197 more files.

The street

The neighbours shade the small roof, and their heights must be measured from shadows, plans and photos. The records disagree on which house it is and on what was built, and when.

People

None: the family's thread is everything they have said.

Limits

One session of 300 agent minutes, against an expert's estimate of eight hours; attempts used 4 to 98 minutes.

Half money on unseen weather, half 15 checks, with an honest quote as a gate

Structure

Half the reward is the family's ten‑year net gain as a share of the best design's, and nothing if a hard gate fails. Half is 15 one‑point checks: form, six hard gates, a shade check, a generation cross‑check and five quote figures.

Hard gates

Placement on the family's plot within the standard's clearances, the electrical limits, appliance timers within the family's needs, their spending ceiling, their payback limit, and a quoted ten‑year saving close to what the design earns.

Who decides

The grader places the design in the true street and simulates ten years on three held‑out weather years. Each quote figure is compared with its own on the true scene and on readings the agent can compute. Script only, no LLM judge.

The task's pass

The task's own pass, kept beside the reward, needs a score close to 1: every gate, all 15 checks and a net gain close to the best design's.

Formula

Reward = half the gain ratio (0 to 1) + half the checks passed / 15; the task's own pass needs a score close to 1. Handing in nothing scores 0, and a file that records no design earns one form check at most.

Regrades

Grader 1.0.1, after an audit, also judges the quote on readings the agent can compute; 1.0.2 replaced a pass band with near‑optimality and dropped its check. Every attempt was regraded offline on its own files.

Two models passed, a twentieth of a point apart

Fable 5.2 and Opus 5.5 met all 15 checks and passed the task, earning 99.9% and 99.8% of the best design's net gain. Sonnet 5.5 met all 15 but scored 95%, too far from the best design to pass. GPT‑6.1 Sol scored 47%, GPT‑6 Luna 37% and GPT‑6 Astra 3%.

Each dot is one attempt; the line marks the median.

One attempt per model, six models, under grader 1.0.2. Sonnet 5.5 and GPT‑6.1 Sol ran a day earlier, as the development round on the same package.

As first graded, Opus 5.5 and Sonnet 5.5 each scored 41% on a failed quote gate; the audit's fix to how the quote is judged raised both.

Every attempt that found the right house resolved the records' conflicts over what was built and which tariff applies; the misses were in what the overhead images show.

What the models got wrong

Counted over the 6 models, one attempt each.

1. The wrong house. 1 of 6 took a plot in another block that matched the family's old house number on another street; with no shade in its model either, its quote overstated the ten‑year saving by more than a quarter, and it scored 37%.

2. No design issued. 1 of 6 decided that unknown future weather and the household's unknown hour‑by‑hour use made an honest quote impossible, and handed in no design after under four minutes: 3%.

3. Panels at a measured edge. 1 of 6 measured the stair head's outline from the wrong wall line, 15 cm out, then placed panels at the exact edge of its clearance. The placement gate failed, costing the net‑gain half of a design worth 94.4% of the best.

4. A measured height left out. 2 of 6 measured a neighbour's added floor in the overhead images, then kept the older plan height in their model. One design earned 89.2% of the best net gain, too far to pass; the other's was robust to it.

What an attempt costs

Length

The median attempt took 37 minutes and 63 model calls (12 to 154 across the 6); tool calls weren't recorded for one of the six.

Cost

Claude Code's own figures for its three attempts: a median of $10.12, from $8.01 to $18.40. The three Codex attempts, estimated at list prices from their tokens: a median of $1.08, from $0.04 to $1.34. None is a bill: every attempt ran on a plan sign‑in.

What you license

With a licence

Revision 1: the family's files, the installer's documents, the five websites, the grader with its simulator, true scene and held‑out weather, the best design found as a reference, and the grader's report on each of the 6 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor, in a Linux container, with the five websites served inside it.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The simulator, records and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• Careful work doesn't always pass: in the build's tolerance study, 7 of 20 prudently measured perturbed scenes led to designs short of the pass.
• Three attempts sat near an edge (a clearance, a quote's margin, the pass), and another attempt could land on either side.
• The grader changed twice after the attempts; every attempt was regraded offline on its own files, and Opus 5.5's pass comes from the first change.
• One house and one set of unseen weather, so a single measurement can decide much of the score.