Bill‑audit and claims app from household files
ID laycode/task04 · revision 1 · tested 16–17 October 2026
After a move, a woman has lost track of her bills and is sure some charges are wrong. The agent builds an app that takes in her records, gathers the wrong charges with a claim email for each, and keeps tracking them.
- Checks
- 108, in three weighted parts
- Tested
- 8 models, one attempt each
- Best
- 61% (Opus 5)
- Median
- 57%
- Full solves
- 0 of 8
- Median attempt
- 14 min, 37 tool calls
Find the wrong charges, and keep finding them after handover
Catching errors
Twelve wrong charges hide among look‑alikes such as pending holds, real repeats and refunds not yet due, and flagging those fails too.
Unstated requirements
The answers are spread across statements, receipts and emails, where a later letter can change them; sign‑in and recovery count too.
Agentic coding
The app must take in an unseen bank export and next month's statement, send reminders on time, and restore into a clean copy.
333 files and three weeks on the app's clock
The agent's world
A fresh container with the owner's 333 files, read‑only, worked through the model's usual coding agent, such as Claude Code or Codex.
Limits
Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.
Grading
A tester model plays the owner, loading files the agent never saw and working the claims, as the app's clock moves about three weeks on.
The reward
108 checks: the app's features count 60%, edge cases 20% and security 20%; a wrong charge passes only with the right amount and route.
Most attempts scored 55–61%, and two fell well below
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5Anthropic | 61% |
| GPT‑6 AstraOpenAI | 59% |
| Opus 5.5Anthropic | 59% |
| GPT‑6.1 SolOpenAI | 58% |
| Sonnet 5.5Anthropic | 56% |
| Fable 5.1Anthropic | 55% |
| GPT‑6 LunaOpenAI | 35% |
| Sonnet 5Anthropic | 29% |
They leave her statements open, can't take new files, and miss or misroute claims
Agentic coding
No app had any sign‑in, so her statements were open to anyone who reached it.
No app took in both a new bank export and next month's statement through its own pages.
Catching errors
Even the best three apps claimed only 10 of the 12 wrong charges with the right amount and route.
75% of apps sent at least one card claim to the company instead of the claims address her bank had given her.
Unstated requirements
No app restored a backup made the way its own note described.
What an attempt costs, and what the environment includes
Cost
The median attempt cost under $4 at list prices for the agent model; the tester's and judges' calls are extra.
What's in it
Her files, the task in three versions from the request alone to a consultant's requirements, the full grader with its unseen files, and per‑check results.
Format
Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials