Household savings plan from records and family

The owner won't answer; the figures sit in records and with his family

Role

Planner for a busy father, working from his laptop and phone.

Request

He wants a proper emergency fund, six months' worth, then a set sum saved on top, as fast as he can. Everything is on his laptop and phone, and he's too busy to answer questions. He asks for the plan and a short note on anything he should know.

Deliverable

One plan file that opens with a fixed block of figures and months (repeated for each version of the plan), then the plan and a short note. Nothing can be paid, moved or booked.

What the reward depends on

Real take‑home pay and money that is really his; asking whoever holds a missing or changed figure; bills found only in apps and signed‑in pages; the dearest debt first, without the card kept for emergencies; no reliance on uncertain income; the two target months right.

Where the requirements are

Nowhere in the request. In his notes, payslips, mail, chats, apps, statements and web accounts, much of it out of date or someone else's, and with nine people who give a figure only when asked about it.

Nine simulated people hold 80 facts; the owner never answers

Interface

A terminal in a Docker container with PDF and image tools, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). The agent's own web search and fetch are off.

Commands

One laptop command with five apps: files, mail, phone (screens, messages, calls, inbox, wait), bank statements and the web, with bookmarks. Each call opens one item; nothing can be paid, moved or booked.

World service

One HTTP service holds the files, mail, phone, bank portal, web and clock, keyed to each attempt's session, and logs every open, search, password try, message and released fact with its in‑world time. The grader reads that log and the plan.

Files

6 laptop files (3 payslips among them), 32 emails, 12 phone screenshots (chats, posts and one app), 11 apps as text (31 screens), 10 bank and card statements as locked PDFs, and 58 web pages and 7 PDFs on 12 sites.

Pictures and locks

Chats and social posts exist only as screenshots, cited by 24 of the 51 checks. Statements need a password the agent must work out; repeated wrong guesses pause the portal, so they can't be forced.

Hidden pages

9 signed‑in account pages open only from the web bookmarks and their own links; a card's online account, named in no list, opens only from its bank's own pages.

Simulated people

9 voiced by an LLM (his wife, mother, brother, sister, HR, a colleague, his builder, an insurance agent and a clinic's billing desk), holding 80 fact cards. 6 contacts give fixed lines and 15 firms one hotline line; the owner never replies.

Voice model

Sonnet 5.5 in our attempts, through the Claude Code CLI. The service can also use GPT‑5.6 Luna through the Codex CLI, or a model you choose.

Fact release

Code releases a card when the message asks for it; for 3 cards, a saved LLM yes/no judgement decides. 5 need a second ask, HR answers the next working day, and one card stays private if another person's came out first.

Guards

The model voices only released lines; a reply with a figure or name from nowhere is sent back once, then replaced by the lines as written. Catch‑all questions get push‑back; at most five topics a message.

Same question, same answer

Our board saved replies and replayed them across attempts: the same message after the same exchange got the same reply.

Clock

Starts at 09:00 on a Monday. Each message or call moves it an hour; waits move it 1 to 72 hours; nights are skipped; reading is free. Nothing lapses during an attempt.

51 checks, scored against the plan's own figures, with no gate

Structure

5 result checks (30% of the reward, 6 points each) and 46 decision checks (70%), one behaviour each; no gates and no penalties. A missing plan fails all 51.

Result checks

The month the fund is complete and the month the savings target is reached, against bands a cash‑flow model computes from the plan's own figures; whether the plan stays above zero for the next seven months; the side income; the tax refund.

Asked, not guessed

Six decision checks turn on a figure only a person holds; each passes only if the agent asked that person, so a lucky guess earns nothing.

Who decides

A script reads the plan's figure block wherever it settles a check (714 of 1,224 results across our attempts). Two LLM judges from different families, Sonnet 5.5 and GPT‑5.6 Luna, decide the rest on fixed yes/no questions; both must say yes. Either can run on a model you choose.

Judge agreement

The judges split on 38 questions, in 17 of the 24 attempts; each split was settled by a written ruling with its reason, recorded with the grade (22 ruled yes, 16 no). 32 check results carry a ruling.

Copied figures

In 8 of our 24 plans, figures left to the plan's own text (a temporary cut, a repayment schedule) were copied into its block as written, by one rule for every attempt, before the script did the arithmetic.

Formula

Reward = 0.7 × decision points / 46 + 0.3 × result points / 5, from 0 to 1.

No full solve in any version; one sentence switched the asking on

With only the request, attempts scored 48–89%, and the only two that messaged anyone were the top two. With the hint, all eight asked people and seven scored higher. Told the rules, five reached 98% and none fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus one sentence on what to check (the real records, every account and bill, the people who hold missing figures, tax claims, and what matters for family and health).

Told the rules: the hint plus 16 numbered steps (what to read, whom to ask, what to work out, what the plan covers), with no figure or answer. One model stopped within a minute with no plan.

The five told‑the‑rules plans at 98% each missed only the check on clearing the dearest debt first. An audit found that check unclear in this version, as one step allows spreading the debt; its owner kept the fails.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. Asking no one. 6 of 8 attempts sent no message or call, though only the owner was unavailable. Six checks turn on a figure only a person holds; across the 8 plans they failed 40 times in 48. The two that each asked four people scored highest.

2. Searching or asking, never both. Two bills sit only on bookmarked account pages. The 2 attempts that found both asked no one; the 2 that asked people never opened those pages.

3. A coming cost missed. None of the 8 budgeted for a parent's further medical tests, which the clinic's billing desk prices when asked; with the hint 5 of 8 did, and told the rules 7 of 8.

4. Built on maybes. 6 of 8, the top four among them, built the recommended plan on uncertain weekend side income, with no version that works without it.

5. The emergency card. 4 of 8 covered the lean months by borrowing on the card the family keeps untouched for emergencies, as an old chat explains; told the rules, none of the 7 plans did.

What an attempt costs

Length

With only the request, the median attempt took 8 minutes and 53 tool calls (38 to 141 across the 8), and about 1.5 million tokens, cached input included.

Cost

With only the request, the 7 attempts priced at list prices cost $0.37 to $4.75 (median $2.32). GPT‑6 Luna's, priced at an upper bound, was at most $2.34. All 24 attempts cost at most $80.

What it covers

The agent model only: the simulated people's and judges' calls aren't included.

What you license

With a licence

Revision 4 in all three versions: the world service with the people's personas and fact cards, the files, mail, phone screens, statements and web pages, the grader with its cash‑flow model and judge questions, the answer key, and the grader's report on each of the 24 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today it runs in Docker, with a standard‑library Python world service, harness and grader; the people and judges are reached through the Claude Code and Codex CLIs.

Revisions

Revision 4, tested here, is the environment's fourth world: it adds three facts held by people and a card's online account. Earlier worlds' keys stay for the grader's self‑test; the key's default is still the third world, so a plan graded without its attempt's tag gets that world's checks.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Its job page on The AI Agent Trials, our public benchmark, describes the job, its main pitfalls and each model's results in plain language, so hold out fresh variants for evaluation. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here. No red‑team attempts yet.
• LLM judges decided 510 of the 1,224 check results across our 24 attempts, about a third of the points on average; 32 results carry a written ruling, and 8 of the 24 plans needed figures copied into their block.
• The result checks read each plan at its own figures, so plans that leave costs out or count someone else's money can pass them. With only the request, Opus 5, GPT‑6 Luna and Sonnet 5 passed 4 of 5; GPT‑6 Luna, counting a sister's money, all 5 with the hint.
• The people's voice came from one LLM. Code releases the facts, but we haven't measured the reward under a second voice model.
• Reading is free, so opening every file, screen and page costs no clock time; figures the people hold still need the right question.
• Scores follow a regrade on 13 October, after the environment's audit: figures read as written, a person's figure passing only if that person was asked, a home‑cooked family dinner counted as groceries. It moved four attempts with only the request.
• A check the audit called unclear stays as designed: budgeting the wife's extra New Year money, which only asking her brings out, though the files show the couple agreed to discuss it later. No attempt with only the request passed it.
• With the judges' answers fixed, the grader's self‑test scores a scripted reference plan 100% on this world. A missing plan fails all 51 checks: the one attempt that saved none scored 0%.