Tea factory rooftop solar and battery design with its business case

A night‑running factory, a changing load, and value measured on years not seen

Role

The solar consultant to a tea factory that processes leaf through the night and shuts for three months a year, with nobody to ask.

Request

Design rooftop solar and a battery that cut the factory's electricity and diesel bills, with the battery always able to run the dryers out when the grid fails, and make the business case to the board.

Deliverable

A design file (panels by roof, mounts, inverters, battery, one or two seasonal controller settings, works) and a one‑page business case with five figures.

What the reward depends on

The design's ten‑year value on three years it never sees, with next season's factory, against the best design the task's search found; every hard rule met; and business‑case figures the design actually earns.

Where the requirements are

In the engagement letter, the client's engineering basis (its simulator in words and formulas, with a catalogue and cost book), and memos that change the answer: a new tariff, net billing, a connection limit, an expansion.

91 files, no simulator to hand, and nobody to ask

Interface

A shell in a Linux container with Python, numpy, pandas, ezdxf, Pillow and exiftool. There is no simulator: an agent that wants numbers builds its own model from the engineering basis.

Network

None. Codex ran with web search off; Claude Code's web tools were on for Fable 5.2, Opus 5.5 and Sonnet 5.5, and no attempt used them.

Files

91, none collected for this purpose: weather downloads, a utility export and bills, building‑management registers, generator and backup‑power logs, a plucking calendar, procedures, four DXF plans, four site photos, memos and quotes.

Data faults

Time zones, gaps, meters that roll over and outages never logged as outages must be found and cleaned before anything is sized on the data.

The factory

It is changing: a line in the history has gone and new equipment is coming, so the recorded load is the wrong one to size on.

Limits

One session of 180 agent minutes, against an expert's estimate of eight hours; attempts used 10 to 31 minutes.

Half money on unseen years, half 18 checks, script only

Structure

Half the reward is the design's ten‑year value as a share of the best design's, and nothing if a hard gate fails. Half is 18 one‑point checks: form, six hard gates, four on what the design shows was read, two cross‑checks, three business‑case figures and one on value.

Who decides

The grader's simulator, the engineering basis as code, runs the design hour by hour over three held‑out years with next season's factory, and recomputes the business case for the same design on the recorded years. Script only, no LLM judge.

The task's pass

The task's own pass, kept beside the reward, needs every hard gate, the dryers' reserve at every hour among them, and a ten‑year value close to the best design's.

Formula

Reward = half the value ratio (0 to 1) + half the checks passed / 18; the task's own pass needs every gate and a ratio near 1. A design with no solar can still pass up to 6 checks, about 17%.

Self‑test

The best design found scores 1 in Harbor, with all 18 checks met; handing in nothing scores 0.

No model passed; the closest design earned 92% of the best value

Fable 5.2 scored 93%, met 17 of 18 checks, and its design earned 92.4% of the best design's ten‑year value. Sonnet 5.5 scored 80%, Opus 5.5 79%, GPT‑6 Astra 75% and GPT‑6.1 Sol 72%; GPT‑6 Luna's design loses money, and it scored 39%.

Each dot is one attempt; the line marks the median.

One attempt per model, six models. Every attempt met all six hard gates and sized the in‑season reserve for the dryers right; the losses were mostly in size, value and the stated figures.

Sonnet 5.5 and GPT‑6.1 Sol ran a day earlier, on a build whose agent‑facing files are identical, and were regraded under the final grader; no score moved more than 0.4 points.

What the models got wrong

Counted over the 6 models, one attempt each.

1. More solar than pays. All 6 attempts built more solar than the best design, 12.5% to 89.1% more, and the extra panels mostly export at a fraction of the import price. 5 of 6 failed the size check.

2. The coming load read high. 4 of 6 built the expanded factory's load above what its own records support, and sized the solar to it; their designs earned 66% to 77% of the best design's value.

3. A value the design doesn't earn. 5 of 6 stated a ten‑year value 25% or more above what the grader computes for the same design, failing that check; a small error in the yearly saving becomes a large one in the value.

4. Settings tuned to the record. 1 of 6, the best attempt, set its controller from its own sweeps of two recorded years; its design earned 92.4% of the best value, and the best design's controller settings alone would have passed it.

5. The money never modelled. 1 of 6 filled the roofs without modelling the money hour by hour and stated a positive ten‑year value for a design that loses money; its value half earned nothing.

What an attempt costs

Length

The median attempt took 21 minutes and about 48 tool calls (14 to 64 across the 6).

Cost

Claude Code's own figures for its three attempts: a median of $5.79, from $2.94 to $11.21. The three Codex attempts, estimated at list prices from their tokens: a median of $0.98, from $0.03 to $3.97. None is a bill: every attempt ran on a plan sign‑in.

What you license

With a licence

Revision 1: the factory's files, the engineering basis, catalogue and cost book, the grader with its simulator and held‑out years, the best design found as a reference, and the grader's report on each of the 6 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor, in a Linux container with no network; the grader needs only Python's standard library.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score, so hold out fresh variants for evaluation. The simulator, records and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• One factory and one set of unseen years, so a single reading of the coming load can decide much of the score.
• The value is measured against the best design the task's own search found and capped there, so a better design outside that search would score no higher.
• Four of the 18 checks test properties the engagement letter's acceptance doesn't list; dropping them changes no attempt's pass and no order.
• Costs are estimates from token counts; no attempt was billed per call.