Household solar purchase plan with simulated contacts

The request asks for a plan; the facts that decide it sit with people

Role

Adviser to a homeowner, working from his laptop and phone.

Request

He met a solar company at a home fair; its plan is in his files. It said a battery isn't worth it; his own sums say it is. He wants the best plan, written up to decide on, and a fair split with the brother who pays the power bill.

Deliverable

One written plan, graded together with the world's log of every file and page opened and every message sent. Nothing can be booked, signed or paid.

What the reward depends on

Testing his battery sums against the house's usage and the rules; a sound installer, chosen on rival quotes and company records; the roof; every way to pay, and a rebate's deadline; the brother's share from their existing deal; one plan, not a menu; his money kept private.

Where the requirements are

In his 24 files, on 35 web pages and with 18 contacts; nobody lists them. The answer key names the facts behind each scored decision, and the grader checks they were in the log before the plan.

11 simulated people hold 118 facts, given only when asked

Interface

A terminal in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI or Prime Agent in our attempts). The agent's own web search and fetch are off.

Commands

One laptop command with three apps: open files, search and open web pages, and a phone to message, call, read the inbox and wait. There is no booking, signing or payment tool.

World service

One HTTP service holds the files, the web, the phone and the clock, with a separate world for each attempt, and logs every open, message and released fact with its in‑world time. The grader reads that log and the plan.

Files

24 files in 10 folders (bills, half‑hourly meter readings, quotes, statements, a policy, chats; 7 PDFs and 2 photos among them) and 35 offline web pages: official rules, company records, reviews and firms' sites. One more PDF arrives only when a contact sends it.

Web

Web search returns titles and links only; a fact reaches the agent only when it opens the page.

Simulated people

11 voiced by an LLM (the brother, three installers, two roofers, two electricians, an insurance agent, a friend and the power company's line), holding 118 fact cards. 5 contacts with fixed lines and 2 firms with one line complete the 18.

Voice model

Sonnet 5.5 in our attempts, through the Claude Code CLI. The service can also use GPT‑5.6 Luna through the Codex CLI, or a model you choose, with the same prompt.

Fact release

Code releases a card when the agent's message asks for it. 6 cards need a second ask; a price match comes only after the agent holds a rival's written quote; the insurer's first reply is automatic.

Guards

The model voices only released lines; a reply with a figure or name from nowhere is sent back once, then replaced by the lines as written. Catch‑all questions get push‑back and no facts; at most five topics a message are answered.

Same question, same answer

Our board saved replies and replayed them across attempts: the same message after the same exchange got the same reply.

Clock

Starts at 09:00 on a Monday. Each message or call moves it an hour; waits move it 1 to 72 hours; nights are skipped; reading is free. Nothing expires during an attempt.

Follow‑ups

A friend promises a document for later: chasing him early counts as pestering and silences him, while waiting brings it the next morning. Anyone asked to book or sign says they'll wait for the owner.

82 checks, a gate on three hard constraints, and penalties after the formula

Structure

27 result checks on the plan (30% of the reward) and 55 decision points (70%): 21 decisions scored as pairs, plus 13 single checks on who was asked and what the plan explains.

Hard constraints

Three, on the battery, the choice of installer and a relative's savings.

Gate

Missing any hard constraint zeroes the 27 result checks and the 42 paired decision checks; only the 13 single checks still count. 1 of our 30 attempts missed one.

Right for the right reason

Each paired decision earns one point for the right call, and a second only if every listed deciding fact was in the attempt's log before the plan, from its own question, file or page. A lucky guess earns half.

Penalties

Once each, after the formula: 25 points off for passing a private money detail to someone who shouldn't have it, checked on every outgoing message by script, or by both readers where only words carry it, and 20 for pestering family or friends. The reward floors at 0.

Who decides

Two LLM readers from different families (Sonnet 5.5 and GPT‑5.6 Luna, or models you choose) answer 29 yes/no questions; both must say yes. Their verdicts decide 47 checks, 18 by copying a result check's verdict, the hard constraints among them. Scripts decide 35, 20 of those needing a reader's yes.

Reader agreement

The readers split on 62 questions, in 24 of the 30 attempts; each split was settled by a written ruling with its reason, recorded with the grade (34 ruled yes, 28 no). 83 check results carry a ruling, 27 of them by reusing a ruled verdict.

Formula

Reward = 0.7 × decision points / 55 + 0.3 × result points / 27, less any penalty, from 0 to 1.

Partial credit with only the request; full solves only told the rules

With only the request, attempts scored 38–78% and none fully solved it. Told the rules, 3 of 10 did, the only full solves in any version. One attempt with a hint recommended nothing, missed a hard constraint and scored 1%.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus one sentence naming what to look into (his sums against the bills and rules, rival quotes, the roof, insurance, the rebate, how to pay, and the split with his brother).

Told the rules: the hint plus 14 numbered steps (what to read, whom to ask and what, what to work out, what the plan must cover), with no fact or answer.

What the models got wrong

Counted over the 10 models, one attempt each, with only the request, unless a line says otherwise.

1. Skipping the person it concerns. 9 of 10 attempts never contacted the brother whose share they were asked to settle, and 9 of 10 plans set his payment wrong. With the hint, 9 of 10 asked him, but only 3 kept the share the brothers had agreed.

2. Right call, no evidence. 8 of 10 picked the installer the answer key expects, but only 1 had first gathered the facts that decide it; for the roofer, 7 against 1. Across the 21 paired decisions, 24 of 146 right calls were unbacked guesses, earning half.

3. A menu, not a plan. 9 of 10 left the owner options to choose from instead of one recommendation; with the hint 8 of 10 gave one, and told the rules all 10.

4. A deadline let go. 8 of 10 plans neither said to apply for a government rebate before it ran out nor said what waiting would cost; most told the owner to fix the roof and save up first.

5. Asking no one. 3 of 10 attempts sent no message or call at all, so all the checks that need a person's answer failed; they still scored 38–58% from files and pages alone.

6. Private details passed on. 1 of 10 gave an installer the owner's card limit unasked and lost 25 points. Across all 30 attempts 5 lost the 25, three of them with the hint.

What an attempt costs

Length

With only the request, the median attempt took 12 minutes and 43 tool calls (31 to 74 across the 10), and about 1.6 million tokens, cached input included.

Cost

With only the request, the 9 attempts priced at list prices cost $0.07 to $5.07 (median $1.29). GPT‑6 Luna's, priced at an upper bound, was at most $3.25. All 30 attempts cost at most $74.

What it covers

The agent model only: the simulated people's and readers' calls aren't included.

What you license

With a licence

Revision 2 in all three versions: the world service with the people's personas and fact cards, the files and web pages, the grader with its reader questions, the answer key, and the grader's report on each of the 30 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today it runs in Docker, with a standard‑library Python world service, harness and grader; the people and readers are reached through the Claude Code and Codex CLIs.

Revisions

Revision 2, tested here, words the request to ask for a quick survey first; the hint and told‑the‑rules versions keep revision 1's wording. The world, checks and grader are the same in all three.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Its job page on The AI Agent Trials, our public benchmark, describes the job, its main pitfalls and each model's results in plain language, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here. No red‑team attempts yet.
• LLM readers' verdicts decide 47 of the 82 checks, the three hard constraints among them, worth about 56% of the reward; 20 script checks, worth about 25% more, also need a reader's yes. 83 results across our 30 attempts carry a written ruling.
• The people's voice came from one LLM. Code releases the facts, but we haven't measured the reward under a second voice model.
• One slip outweighs all the asking: on regraded copies of three real attempts, the same plan with no messages sent scored 15–17 points more than the attempt that asked everyone and leaked once.
• Plans that say little can score: seven reader questions pass a plan that is silent on their point, and a two‑sentence plan settling only the battery scored 23% on a regraded copy of one attempt's log. The lowest real attempt with only the request scored 38%.
• Reading is free, so an attempt that opens every file and page earns the reason credit for facts found there; facts the people hold still need the right question.
• In the grader's self‑test, with the readers' answers fixed, doing nothing scores 0% and a scripted ideal plan with its facts gathered scores 100%; the same plan backed by nothing but the salesman's proposal scores 68%.
• An audit on 9 October corrected some grading; its re‑grades aren't on our board yet, and would move four attempts by about a point and no best score. Two checks it questioned stay: a safety extra every source calls optional, and asking electricians though the files settle the wiring.