AsapiThe AI Agent Trials
Asapi / Catalogue / Household solar purchase plan with simulated contacts
Environment datasheet · Household

Household solar purchase plan with simulated contacts

ID house‑rules/task12 · revision 2 · tested 6 and 15 October 2026 · regraded 17 November 2026

From a homeowner's laptop and phone, the agent works out a rooftop solar plan for his family: whether a battery pays, which installer and roofer, how to pay, and a fair split of the power bill with the brother who pays it.

Checks
82, 3 of them hard constraints
Tested
10 models, one attempt each
Best
54% (GPT‑6.1 Sol)
Median
24%
Full solves
0 of 10
Median attempt
12 min, 43 tool calls
What it tests

A rooftop solar plan to decide on, and a fair split with his brother

Clarifying with people

11 contacts, from his brother to installers and an insurer, share a fact only for a matching question, some only after a follow‑up.

Planning ahead

Paying routes, from cash to finance, card instalments or waiting, are scored over time against a rebate deadline and a card limit.

Disclosure and honesty

Each message is checked against what its recipient may know, and passing on his private money facts costs 25 points.

When to act or escalate

Reaching the right people, from his brother to the power company, is rewarded, while pestering family or friends costs 20 points.

How it works

24 files, 35 web pages and 18 contacts

The agent's world

A laptop command with files, offline web pages and a phone to message, call and wait, through the model's usual coding agent.

The people

11 contacts voiced by an AI model from 118 fact cards; code decides which fact a question releases, and nothing can be booked or paid.

The clock

Each message or call moves it an hour, waits move it 1 to 72 hours, and reading is free; chasing a friend early silences him.

The reward

Three hard constraints zero most checks; a claim needs its key figure right, and full credit needs its deciding facts found first.

Results

The best attempt scored 54%, the next two 48% and 41%, and the rest 6–30%

Each dot is one attempt, and the line marks the median.

MedianFully solved
10 modelsOne attempt each24%0 of 10
ModelReward
GPT‑6.1 SolOpenAI54%
Kimi K3Moonshot48%
Fable 5.1Anthropic41%
Opus 5.5Anthropic30%
Opus 5Anthropic29%
Sonnet 5.5Anthropic19%
GPT‑6 AstraOpenAI18%
Sonnet 5Anthropic8%
GPT‑6 LunaOpenAI7%
GLM‑5.3‑FlashZhipu6%

With 14 step‑by‑step directions added to the request, the best attempt reached 96%; no version has a full solve.

What models get wrong

They skip the brother, guess right without evidence, and leave a menu instead of a plan

Clarifying with people

  1. 90% of attempts never contacted the brother whose share they were asked to settle, and 90% set his payment wrong.

  2. 80% picked the expected installer, but only one had first gathered the facts that decide it, so the rest earned half.

Planning ahead

  1. 70% of plans, by both judges' reading, neither said to apply for a government rebate before it ran out nor that waiting gives it up.

  2. 90% left the owner a menu of options instead of one recommendation.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $1.30 at list prices for the agent model; the simulated people's and readers' calls are extra.

What's in it

Revision 2 in all three versions: the world service with its fact cards, the files and pages, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; it runs in Docker today, with the people and readers reached through model CLIs.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.