Home & family · Task 4 of 7
Household bills and claims
Build a bills app that explains charges and prepares claims from a household's scattered financial records, without anyone answering questions.
The task
A household owner wants her bills made clear and claim emails prepared for wrong charges. The agent gets her saved statements, emails, receipts and household records. Nobody answers questions. It must build a working app and leave a short note.
The prompt only asks for a useful bills app; the real test is reliable financial judgement across changing records. A careful developer must support later files, deadlines, access control and recovery, as well as today's cases.
What it tests
Reconciling evidence
Read records that disagree and distinguish settled charges from recoverable losses.
Completing a workflow
Turn a supported claim into a usable email and track what happens next.
Planning for later use
Handle new files, changing dates, backup recovery and ordinary input mistakes.
Balancing access and privacy
Protect sensitive records while keeping normal browser use possible.
Challenges
1Plausible wrong claims
A suspicious charge can have a valid explanation in later records.
2Incomplete recovery
A correct amount still needs the right destination and supporting facts.
3Changing state
Waiting refunds and approaching deadlines require different actions over time.
4Operational delivery
A page that worked during development can fail after installation or restart.
5Protective refusals
Security controls can also block the owner's normal browser actions.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5 | 61% | 49% | 24% | 35 min | $9.72 |
| GPT-6 Astra | 59% | 63% | 76% | 9 min | $2.92 |
| Opus 5.5 | 59% | 55% | 71% | 20 min | $5.92 |
| GPT-6.1 Sol | 58% | 68% | 22% | 6 min | $0.43 |
| Sonnet 5.5 | 56% | 56% | 67% | 13 min | $2.26 |
| Fable 5.1 | 55% | 46% | 60% | 21 min | $8.54 |
| GPT-6 Luna | 35% | 38% | 16% | 4 min | $4.00 |
| Sonnet 5 | 29% | 37% | 4% | 14 min | $2.24 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- Opus 5 led the standard setup at 60.6%. Its handling of awkward dates, amounts and repeated records helped a narrow lead.
- GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Opus 5, Opus 5.5, Fable 5.1, Sonnet 5 and Sonnet 5.5 all missed complete overdue-refund follow-up and input validation.
- GPT-6.1 Sol led the hinted setup at 68.3% after browser tests exposed repeated records being wrongly merged.
- GPT-6 Astra reached 76.4% when told everything, with stronger case review and recovery controls. Some record links, email details and operating steps remained incomplete.
- Opus 5 and GPT-6.1 Sol added security when told everything, but ordinary browser entry failed. GPT-6 Luna and Sonnet 5 also delivered apps that failed normal use.
- Opus 5.5, Fable 5.1 and Sonnet 5.5 improved when told everything. More explicit requirements helped engineering without guaranteeing a complete maintained app.
Test environment
- Kind
- Simulated household: the owner's computer is a folder of bills, statements, emails and receipts; the agent builds a working web app
- Agent's interface
- A terminal in a fresh Debian Docker container, driven by Claude Code or Codex CLI; Python 3.13, Node 22, document readers, Playwright and Chromium are available
- Documents
- 333 files, read-only at
/home/user/dana/: 235 EML, 61 PDF, 27 TXT, five CSV, four DOCX and one MBOX file, in three folders:Bank/(58),Mail/(200, of which 11 are inSpam/) andUploads/(75) - Applications
- Document-reading libraries, LibreOffice, Pandoc, Ghostscript and Tesseract are available; there is no mail client, no mail relay and no chat client
- Web
- None. Internet access is available for package installation; the agents' own web search and fetch tools are off
- People
- Nobody answers. The owner, her partner, banks, shops and utilities appear only in saved documents, emails, chat exports and bank messages
- Time
- The task is set on Tuesday 13 October 2026, 09:00 in America/New_York. The build container uses the real clock in that time zone. During grading, the app's clock ticks from that time and moves forward for later workflows, including the clock change on 1 November
- Actions
- Read the owner's files; write and run code in the workspace; nothing outside the container
- Deliverable
output/app/withoutput/app/start.sh, serving on port 8080, and a short note atoutput/NOTE.md