AsapiThe AI Agent Trials
Asapi / Catalogue / Bill‑audit and claims app from household files
Environment datasheet · Household

Bill‑audit and claims app from household files

ID laycode/task04 · revision 1 · tested 16–17 October 2026

After a move, a woman has lost track of her bills and is sure some charges are wrong. The agent builds an app that takes in her records, gathers the wrong charges with a claim email for each, and keeps tracking them.

Checks
108, in three weighted parts
Tested
8 models, one attempt each
Best
61% (Opus 5)
Median
57%
Full solves
0 of 8
Median attempt
14 min, 37 tool calls
What it tests

Find the wrong charges, and keep finding them after handover

Catching errors

Twelve wrong charges hide among look‑alikes such as pending holds, real repeats and refunds not yet due, and flagging those fails too.

Unstated requirements

The answers are spread across statements, receipts and emails, where a later letter can change them; sign‑in and recovery count too.

Agentic coding

The app must take in an unseen bank export and next month's statement, send reminders on time, and restore into a clean copy.

How it works

333 files and three weeks on the app's clock

The agent's world

A fresh container with the owner's 333 files, read‑only, worked through the model's usual coding agent, such as Claude Code or Codex.

Limits

Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.

Grading

A tester model plays the owner, loading files the agent never saw and working the claims, as the app's clock moves about three weeks on.

The reward

108 checks: the app's features count 60%, edge cases 20% and security 20%; a wrong charge passes only with the right amount and route.

Results

Most attempts scored 55–61%, and two fell well below

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 modelsOne attempt each57%0 of 8
ModelReward
Opus 5Anthropic61%
GPT‑6 AstraOpenAI59%
Opus 5.5Anthropic59%
GPT‑6.1 SolOpenAI58%
Sonnet 5.5Anthropic56%
Fable 5.1Anthropic55%
GPT‑6 LunaOpenAI35%
Sonnet 5Anthropic29%
What models get wrong

They leave her statements open, can't take new files, and miss or misroute claims

Agentic coding

  1. No app had any sign‑in, so her statements were open to anyone who reached it.

  2. No app took in both a new bank export and next month's statement through its own pages.

Catching errors

  1. Even the best three apps claimed only 10 of the 12 wrong charges with the right amount and route.

  2. 75% of apps sent at least one card claim to the company instead of the claims address her bank had given her.

Unstated requirements

  1. No app restored a backup made the way its own note described.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost under $4 at list prices for the agent model; the tester's and judges' calls are extra.

What's in it

Her files, the task in three versions from the request alone to a consultant's requirements, the full grader with its unseen files, and per‑check results.

Format

Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.