Bill‑audit and claims app from household files

Find the wrong charges, and keep finding them after handover

Role

Builder of a personal bills app for a household owner who is away all week, working from a read‑only copy of her files.

Request

Her bills have been a mess since the move. She wants an app that takes in her bills, statements and receipts, shows the wrong charges in one place with a claim email for each, ready to paste, and tracks them after.

Deliverable

A running app and a short note. The app must also take in files that arrive after handover: a new bank export and next month's statement.

What the reward depends on

Reading every record; claiming each wrong charge for the right amount by the right route; leaving look‑alikes and refunds not yet due alone; deadlines; tracking claims; sign‑in, recovery and a note.

Where the requirements are

Spread across statements, receipts, emails and bank messages, where a later letter can change the answer. Sign‑in and recovery are never asked for. A script confirms the request avoids the checks' subjects, such as deadlines, refunds, logins and backups.

333 files, then next month's statement

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

333, read‑only: 235 emails, 61 PDFs, 27 text files, 5 CSV exports, 4 Word files and a mailbox, in bank, mail (spam included) and upload folders.

What's in them

12 wrong charges, 8 look‑alikes that are fine, and 7 items that are settled, not yet due or past their time limit, across shops, subscriptions, a phone company, a power company, an insurer and her bank.

Limits

120 minutes and 150 model turns per attempt; a $10 list‑price spend cap for 22 of the 24, $20 for two told‑the‑rules attempts.

Grading copy

The output is copied into a fresh container without installed packages, so the app must set itself up, with her files mounted again.

Tester

Sonnet 5.5 plays the owner in fresh browser sessions: loads a bank export the agent never saw, opens and copies claim emails, raises, sends, resolves and dismisses claims. Page actions only.

Clock walk

The app's clock moves about three weeks ahead, across a daylight‑saving change, while next month's statement arrives and a mail sink records reminders.

Probes

Hidden edge files, security probes, a restart, offline use, a moved folder, recovery into a clean copy, a burst of simultaneous requests and a password reset.

108 checks in three sections, 95 decided by code

Structure

Build: 84 checks share 60 points (reading the files, judging each charge, claims and follow‑up, running and handover). Edge cases: 10 share 20. Security: 14 share 20. No gates.

Formula

Reward = (60 × build passed / 84 + 20 × edge passed / 10 + 20 × security passed / 14) / 100. A check that doesn't apply counts as passed.

Strict claims

A wrong charge passes only if it sits in the wrong‑charges place with the right claim amount and the right route, to the bank or to the company. Flagging a look‑alike or a refund not yet due fails its own check.

Who decides

Scripts decide 95 checks; two LLM judges, Sonnet 5.5 and GPT‑5.6 Luna, decide 13 and must agree. Either can run on a model you choose.

Judge agreement

Where both verdicts are kept, the judges disagreed on 30 of 206 judged results across the 24 attempts.

Rulings

256 of the 2,592 results (10%) carry a written ruling by the environment's author: the 30 splits, 19 where both judges agreed and 207 script results. A review on 19 October changed 81 decisions in 12 attempts, 80 to passes, before publication.

Kept apart

Only her files enter the agent's container: no key, grader or reference. A script rejects added lines that give a figure, a listed name from her files or a check id.

Reference solution

None stored. The audit notes the development reference app was incomplete. No attempt fully solved it; the best scored 76%, told the rules.

Told the rules, four of eight fell below 25%; none fully solved it

With only the request, the 8 attempts scored 29–61%; with a hint, the best reached 68%. Told the rules, four of 8 rose to 60–76% and four fell below 25%, two of them turning away the owner's own first sign‑in. None of the 24 fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus five plain lines from a friend on what matters most.

Told the rules: the request plus 13 requirements from an IT consultant, which also ask for a sign‑in and backups. They give no facts from her files.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. Statements open to anyone. 0 of 8 apps hid her data from someone not logged in, or had any sign‑in at all; told the rules, 7 of 8 hid it.

2. Today's list, not an app. 0 of 8 took in both a new bank export and next month's statement through the app; with the hint, 3 of 8 did.

3. Wrong charges missed. The best three apps found 10 of the 12 wrong charges with the right amount and route; the other five found 0 to 9. Told the rules, GPT‑6 Astra found all 12.

4. Claims to the wrong desk. 6 of 8 apps sent at least one card claim to the company instead of the claims address her bank had given her; 2 of 8 sent every one to the bank.

5. No recovery. 0 of 8 restored a backup made the note's way; told the rules, 2 of 8 did.

What an attempt costs

Length

With only the request, the median attempt took 14 minutes and 37 tool calls (17 to 76 across the 8), and about 3.2 million tokens, cached input included.

Cost

The median attempt cost $3.46 at list prices, from $0.43 to $9.72 across the 8; all 24 attempts came to $149. GPT‑6 Luna has no published price, so its figures, and with them this median and total, are upper bounds.

What's counted

The agent model only: the tester's and judges' calls aren't included.

What you license

With a licence

All three versions, the 333 files, the grader with its tester steps, unseen statement and export, probes and judge prompts, the answer key, and the grader's per‑check results for each of the 24 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser and the tester and judge models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution and no empty‑app test. An attempt whose app never started still met 5 of 108 checks, mostly on its note, for a reward of 4%.
• Four checks were passed by no attempt in any version, among them a reminder at the right hour after the clocks change and a late refund shown as late.
• 13 checks go to LLM judges and 256 results carry a written ruling; the board's scores include the 19 October review's 81 changes.
• The tester is an LLM; we haven't measured the reward under another tester or judge pair.
• No red‑team attempts yet.