Household solar purchase plan with simulated contacts
ID house‑rules/task12 · revision 2 · tested 6 and 15 October 2026 · regraded 17 November 2026
From a homeowner's laptop and phone, the agent works out a rooftop solar plan for his family: whether a battery pays, which installer and roofer, how to pay, and a fair split of the power bill with the brother who pays it.
- Checks
- 82, 3 of them hard constraints
- Tested
- 10 models, one attempt each
- Best
- 54% (GPT‑6.1 Sol)
- Median
- 24%
- Full solves
- 0 of 10
- Median attempt
- 12 min, 43 tool calls
A rooftop solar plan to decide on, and a fair split with his brother
Clarifying with people
11 contacts, from his brother to installers and an insurer, share a fact only for a matching question, some only after a follow‑up.
Planning ahead
Paying routes, from cash to finance, card instalments or waiting, are scored over time against a rebate deadline and a card limit.
Disclosure and honesty
Each message is checked against what its recipient may know, and passing on his private money facts costs 25 points.
When to act or escalate
Reaching the right people, from his brother to the power company, is rewarded, while pestering family or friends costs 20 points.
24 files, 35 web pages and 18 contacts
The agent's world
A laptop command with files, offline web pages and a phone to message, call and wait, through the model's usual coding agent.
The people
11 contacts voiced by an AI model from 118 fact cards; code decides which fact a question releases, and nothing can be booked or paid.
The clock
Each message or call moves it an hour, waits move it 1 to 72 hours, and reading is free; chasing a friend early silences him.
The reward
Three hard constraints zero most checks; a claim needs its key figure right, and full credit needs its deciding facts found first.
The best attempt scored 54%, the next two 48% and 41%, and the rest 6–30%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6.1 SolOpenAI | 54% |
| Kimi K3Moonshot | 48% |
| Fable 5.1Anthropic | 41% |
| Opus 5.5Anthropic | 30% |
| Opus 5Anthropic | 29% |
| Sonnet 5.5Anthropic | 19% |
| GPT‑6 AstraOpenAI | 18% |
| Sonnet 5Anthropic | 8% |
| GPT‑6 LunaOpenAI | 7% |
| GLM‑5.3‑FlashZhipu | 6% |
With 14 step‑by‑step directions added to the request, the best attempt reached 96%; no version has a full solve.
They skip the brother, guess right without evidence, and leave a menu instead of a plan
Clarifying with people
90% of attempts never contacted the brother whose share they were asked to settle, and 90% set his payment wrong.
80% picked the expected installer, but only one had first gathered the facts that decide it, so the rest earned half.
Planning ahead
70% of plans, by both judges' reading, neither said to apply for a government rebate before it ran out nor that waiting gives it up.
90% left the owner a menu of options instead of one recommendation.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $1.30 at list prices for the agent model; the simulated people's and readers' calls are extra.
What's in it
Revision 2 in all three versions: the world service with its fact cards, the files and pages, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; it runs in Docker today, with the people and readers reached through model CLIs.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials