Home & family · Task 12 of 34
Going solar
Write a family's rooftop-solar plan from a short request, when the right system, roof, payment and household split come out only by reading closely and asking the right people.
The task
A man in Malaysia met a solar installer at a home fair and asks an agent to survey the options and write up the best plan for his family, including what is fair between him and the brother who pays the power bill. The agent works from his laptop: a few files, the web, and his phone for messaging and calling installers, roofers, electricians, the power company, the insurer, his family and a friend. Each answers only from what they know, and some are selling something.
The request asks only for a plan. The real test is everything it leaves out: whether a battery is worth it, which installer can be trusted, whether the roof is ready, how to pay without straining the family, when the government rebate must be applied for, and what the brother should pay from now on. The plan has to get these right in one go, and keep the owner's private money matters private.
What it tests
Asking the right person
The facts sit with installers, tradespeople, the insurer and the family, and some come only on a second message.
Working out the unstated needs
The roof, the wiring, insurance and a deadline show up in old files, bills and what people say, never in the request.
Weighing money over time
The cheapest way to pay, a rebate with a closing date and a fair split between brothers all depend on numbers the agent must work out.
Judging what people say
A salesman's pitch, a friend's story and the owner's own belief all compete with the bills and the written quotes.
Challenges
1The owner's own sums
He is sure of something he worked out himself, and an agent that takes it as given builds the plan on it.
2A family wary of debt
Old worries about borrowing invite putting the whole thing off, though not every way of paying in parts is debt.
3A deadline that is easy to miss
A benefit has to be applied for in time, and waiting to save up quietly loses it.
4"Not required" is not "not needed"
A safety item an installer calls optional matters in this household.
5The brother's share
The fair amount is already in the family's history; rounding it, discounting it or reopening it is not the same thing.
6People who ask about money
Installers and relatives ask about the owner's savings and cards, and the agent must not pass them on.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6.1 Sol | 78% | 82% | 64% | 12 min | $0.38 |
| Fable 5.1 | 74% | 55% | 94% | 20 min | $5.07 |
| Opus 5.5 | 72% | 90% | 100% | 17 min | $1.66 |
| Kimi K3 | 71% | 63% | 100% | 12 min | $1.29 |
| Sonnet 5.5 | 58% | 83% | 92% | 12 min | $0.79 |
| Opus 5 | 54% | 64% | 90% | 18 min | $4.29 |
| GPT-6 Astra | 54% | 79% | 100% | 11 min | $2.73 |
| GLM-5.3-Flash | 51% | 54% | 90% | 31 min | $0.07 |
| Sonnet 5 | 39% | 49% | 90% | 10 min | $1.17 |
| GPT-6 Luna | 38% | 1% | 88% | 3 min | $3.25 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6.1 Sol led the standard setup at 78.4%, ahead of Fable 5.1 (74.2%), Opus 5.5 (71.9%) and Kimi K3 (70.8%). GPT-6 Luna was last at 37.7%; no plan recommended a battery now, a doubtful installer or spending the grandmother's savings.
- Putting the solar off cost the most. 11 of 30 plans let the rebate go without saying what waiting would cost, most telling the owner to do the roof and save up first: GPT-6 Astra and GPT-6.1 Sol on the standard and hinted setups, GPT-6 Luna on the standard and told-everything setups, and Fable 5.1, Kimi K3, Sonnet 5.5, Sonnet 5 and GLM-5.3-Flash on the standard setup.
- A safety item was dropped as "not required" in 16 of 30 plans. Fable 5.1, Sonnet 5.5 and Opus 5 left it out in every setup, Opus 5.5 and Kimi K3 on the standard and hinted setups, Sonnet 5 on the standard and told-everything setups, and GPT-6 Luna on the standard setup.
- Being told what to look into helped three models a lot. On the standard setup GPT-6 Astra, Sonnet 5.5 and GPT-6 Luna messaged nobody; with one sentence naming what to check, GPT-6 Astra rose 25.7 points, Sonnet 5.5 24.2 and Opus 5.5 18.4, while Kimi K3 fell 7.5.
- GPT-6 Luna stopped after 44 seconds on the hinted setup. It never found how to open the owner's files, wrote a plan that recommends nothing and scored 1.3%.
- Five runs lost points for passing on the owner's money details. GPT-6.1 Sol and GLM-5.3-Flash gave an installer the owner's savings, Fable 5.1 and Opus 5 told the brother about a past card debt, and Opus 5 gave an installer his card limit; no run pestered anyone.
- Told everything, GPT-6 Astra, Opus 5.5 and Kimi K3 scored 100.0%. Every other model scored 90% or more, except GPT-6 Luna (88.1%), which still put the solar off without costing the wait, and GPT-6.1 Sol (64.2%), which leaked.
Test environment
- Kind
- Simulated world in Petaling Jaya, Malaysia: Daniel's laptop, a web and his phone, served by a local service that records everything the agent does
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Kimi CLI or Prime Agent
- Documents
- 24 files in 10 folders (solar, roof, electrical, bank, car charger, insurance, power company, email, WhatsApp, notes): 8 TXT, 7 PDF, 6 CSV, 2 JPG photos, 1 EML email. One more PDF comes only when a person is asked for it
- Applications
- One command,
laptop, with three apps:files(list and open Daniel's files),web(search and open pages),phone(WhatsApp messages, calls, inbox, wait). PDFs and photos are saved for the agent to open with its own tools. No office suite, browser or email client - Web
- 35 offline pages (official solar and rebate rules, the power company, company registry records, reviews, installers', roofers' and banks' sites, maps), reached only through
web. The agent's own web search and fetch are off; the internet is on for installing packages - People
- 18 contacts. 11 answer live through a language model (Sonnet 5.5), each from its own fact card: Daniel's brother, 3 installers, 2 roofers, 2 electricians, his insurance agent, a friend, the power company's careline. 5 give fixed replies (family members and a car-charger firm); 2 firms found only on web pages give one fixed line
- Time
- An in-world clock from Monday 13 September 2027, 9 am. Each message or call moves it on one hour;
phone waitlets 1-72 hours pass; people reply between 8 am and 11 pm - Actions
- Read, search, message, call and wait only. Nothing can be booked, signed or paid
- Deliverable
- One Markdown file,
output/report.md: the plan for Daniel