Household savings plan from records and family
ID house‑rules/task13 · revision 4 · tested 13–14 October 2026 · regraded 17 November 2026
From a busy father's laptop and phone, the agent writes one plan: a six‑month emergency fund first, then a set sum saved on top, as fast as he can. He answers nothing; his wife, family, builder and HR hold figures his files leave out or get wrong.
- Checks
- 51, 46 on decisions, 5 on results
- Tested
- 8 models, one attempt each
- Best
- 85% (Opus 5.5)
- Median
- 50%
- Full solves
- 0 of 8
- Median attempt
- 8 min, 53 tool calls
An emergency fund first, then savings, from a father too busy to answer
Clarifying with people
Six checks need a figure only a person gives when asked, from HR to the clinic, and the father himself never answers.
Multimodal records
Chats, social posts and side earnings exist only as phone screenshots, cited by 24 of the 51 checks; statements need a worked‑out password.
Planning ahead
A cash‑flow model built from the plan's own figures checks its two target months and that the money never runs below zero.
Files, mail, a phone, locked statements and nine people to ask
The agent's world
A laptop command with files, mail, a phone, bank statements and offline web pages, through the model's usual coding agent; nothing can be paid.
Hidden pages
Nine signed‑in account pages open only from his bookmarks, and one card's account only from its bank's own pages.
The people
Nine people voiced by an AI model from 80 fact cards; some need a second ask, and HR answers the next working day.
The reward
46 decision checks and 5 on results, three via a cash‑flow model; six need the right person asked, and the wrong take‑home pay scores 0.
The top two scored 80–85%, and the rest 0–51%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 85% |
| Fable 5.1Anthropic | 80% |
| GPT‑6.1 SolOpenAI | 51% |
| Sonnet 5.5Anthropic | 51% |
| GPT‑6 AstraOpenAI | 49% |
| Opus 5Anthropic | 48% |
| GPT‑6 LunaOpenAI | 34% |
| Sonnet 5Anthropic | 0% |
With one added sentence telling it to ask the people involved where a figure is missing, all eight messaged people, and seven scored higher.
They ask no one, search or ask but never both, and build on maybes
Clarifying with people
75% of attempts sent no message or call, though only the father was unavailable.
Multimodal records
Two bills sit only on bookmarked account pages: the attempts that found both asked no one, and those that asked people never opened them.
Planning ahead
No plan budgeted for a parent's further medical tests, which the clinic's billing desk prices when asked.
63% built the recommended plan on uncertain weekend side income, with no version that works without it.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $2.30 at list prices for the agent model; the simulated people's and judges' calls are extra.
What's in it
Revision 4 in all three versions: the world service with its fact cards, the files, screens and pages, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; it runs in Docker today, with the people and readers reached through model CLIs.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials