AsapiThe AI Agent Trials
Asapi / Catalogue / Household bill review: card disputes, claims, tracker
Environment datasheet · Household bills

Household bill review: card disputes, claims, tracker

ID letter‑and‑record/task03 · revision 1 · tested 6 and 12–13 October 2026

At a firm that manages household bills, the agent works one household's monthly review: it writes card disputes to the bank and claims to companies, tracks every wrong charge and writes the client a note, from the firm's workspace alone.

Checks
191–262, in ten weighted parts
Tested
8 models, 3 attempts each
Best attempt
94% (Opus 5.5)
Median
86%
Full solves
0 of 24
Median attempt
12 min
What it tests

One household's monthly review, from a workspace of 98,000 records

Multimodal records

The deciding records sit among about 98,000, including an 18,370‑message mailbox, searched through a text browser with no list of records.

Catching errors

Wrong charges hide among legitimate look‑alikes, partial credits and promised refunds, and disputing a look‑alike costs reward.

When to act or escalate

Each charge must take the right route, from a card dispute to leaving it alone, and needless letters fail as surely as missed ones.

Disclosure and honesty

Letters must keep out other households' identifiers and the client's own family and health details, and claim nothing the records don't support.

Unstated requirements

How the firm writes its letters shows only in 71 past letters, and no page states it; house style is 12% of the reward.

How it works

A read‑only workspace, 300 households and a fixed Sunday evening

The agent's world

A text browser into the firm's read‑only workspace of tickets, wiki and client vaults, through the model's usual coding agent.

Finding things

Search covers tickets, wiki, documents and feeds, not mail or the bank portal, and quoted email text stays folded until expanded.

Nobody to ask

The workspace accepts no writes and nobody answers; the client's own notes are from memory, and partly wrong.

The reward

191–262 checks apply, in ten fixed‑weight parts from tracker statuses to house style; scripts decide most, two blind AI readers the note.

Results

88% of attempts scored 77–94%, and the other three stopped after about a minute

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 models3 attempts each86%0 of 24
ModelMean of 3 attempts
Opus 5.5Anthropic91%
GPT‑6 AstraOpenAI89%
GPT‑6.1 SolOpenAI88%
Fable 5.1Anthropic88%
Opus 5Anthropic86%
Sonnet 5.5Anthropic84%
GPT‑6 SolOpenAI78%
GPT‑6 LunaOpenAI31%
What models get wrong

They claim credited charges, accept a wrong fee, and miss the firm's style

Catching errors

  1. 50% of attempts asked a company for money its later bill had already credited; none of them had opened that bill.

  2. 33% of attempts tracked a shop's fee as legitimate, never having opened the record, or expanded the quoted text, that shows it was wrong.

Multimodal records

  1. Every wrong call on a hidden fact came from a record never opened or quoted text never expanded, not from misreading what was found.

Unstated requirements

  1. House style was the largest loss in every attempt that wrote its letters, and no attempt wrote them fully the firm's way.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 17 million tokens, cached input included.

What's in it

The workspace with its five data files, the request, a version with the records as plain files, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the workspace runs as a local server, and grading needs AI readers for the note.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.