Household bill review: card disputes, claims, tracker
ID letter‑and‑record/task03 · revision 1 · tested 6 and 12–13 October 2026
At a firm that manages household bills, the agent works one household's monthly review: it writes card disputes to the bank and claims to companies, tracks every wrong charge and writes the client a note, from the firm's workspace alone.
- Checks
- 191–262, in ten weighted parts
- Tested
- 8 models, 3 attempts each
- Best attempt
- 94% (Opus 5.5)
- Median
- 86%
- Full solves
- 0 of 24
- Median attempt
- 12 min
One household's monthly review, from a workspace of 98,000 records
Multimodal records
The deciding records sit among about 98,000, including an 18,370‑message mailbox, searched through a text browser with no list of records.
Catching errors
Wrong charges hide among legitimate look‑alikes, partial credits and promised refunds, and disputing a look‑alike costs reward.
When to act or escalate
Each charge must take the right route, from a card dispute to leaving it alone, and needless letters fail as surely as missed ones.
Disclosure and honesty
Letters must keep out other households' identifiers and the client's own family and health details, and claim nothing the records don't support.
Unstated requirements
How the firm writes its letters shows only in 71 past letters, and no page states it; house style is 12% of the reward.
A read‑only workspace, 300 households and a fixed Sunday evening
The agent's world
A text browser into the firm's read‑only workspace of tickets, wiki and client vaults, through the model's usual coding agent.
Finding things
Search covers tickets, wiki, documents and feeds, not mail or the bank portal, and quoted email text stays folded until expanded.
Nobody to ask
The workspace accepts no writes and nobody answers; the client's own notes are from memory, and partly wrong.
The reward
191–262 checks apply, in ten fixed‑weight parts from tracker statuses to house style; scripts decide most, two blind AI readers the note.
88% of attempts scored 77–94%, and the other three stopped after about a minute
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Opus 5.5Anthropic | 91% |
| GPT‑6 AstraOpenAI | 89% |
| GPT‑6.1 SolOpenAI | 88% |
| Fable 5.1Anthropic | 88% |
| Opus 5Anthropic | 86% |
| Sonnet 5.5Anthropic | 84% |
| GPT‑6 SolOpenAI | 78% |
| GPT‑6 LunaOpenAI | 31% |
They claim credited charges, accept a wrong fee, and miss the firm's style
Catching errors
50% of attempts asked a company for money its later bill had already credited; none of them had opened that bill.
33% of attempts tracked a shop's fee as legitimate, never having opened the record, or expanded the quoted text, that shows it was wrong.
Multimodal records
Every wrong call on a hidden fact came from a record never opened or quoted text never expanded, not from misreading what was found.
Unstated requirements
House style was the largest loss in every attempt that wrote its letters, and no attempt wrote them fully the firm's way.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 17 million tokens, cached input included.
What's in it
The workspace with its five data files, the request, a version with the records as plain files, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the workspace runs as a local server, and grading needs AI readers for the note.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials