Household bill review: card disputes, claims, tracker

A monthly bill review where nothing says what's wrong

Role

The review desk of a household bill‑management firm, covering an absent account manager on a Sunday evening.

Request

Work one ticket, a household's monthly bill review: write any letters, a tracker and a note to the client as files, in the formats given (letter headers, tracker columns and statuses, word caps). It says nothing about what, if anything, is wrong.

Deliverable

One letter per company still owing money (card disputes to the bank, claims to companies), each with To, Re and Amount lines and at most 350 words; a tracker CSV; and a note of at most 1,750 words.

What the reward depends on

Finding every wrong charge and routing it (dispute, claim, wait for a promised refund, too late, or leave alone) with the right amount and deadline under the current rule; leaving look‑alikes alone; letters that keep out others' details; the house style.

Where the requirements are

The playbook gives the routes; how letters are written shows only in 71 past letters. Six deciding facts sit off the obvious path, in places such as an attachment, a quoted email or a chat transcript, and the client's own notes are memory, partly wrong.

98,093 records for 300 households, read one text page at a time

Interface

A text browser into the firm's read‑only workspace, over MCP, on one Mac: Claude Code 2.1.292 for the Anthropic models and Codex CLI 0.162.1 for the GPT models, each in a sandbox. No shell, no web.

Desk tools

Nine: open, click, read, scroll, back, type, submit, a calculator and a word counter. A screen is a page's title, path and up to 60 lines of text.

Workspace

Tickets with a query navigator capped at 1,000 results; a wiki with page history; and a vault per client with uploads, a mailbox, a view‑only bank portal and bank feeds. Every file shows as text.

Scale

98,093 records for 300 households, from five data files: 7,183 tickets and 679 wiki pages, and for this client 75 uploads, a mailbox of 18,370 messages, 1,201 card transactions and her bank's statements, documents, messages and feed.

Finding things

No list of records. The header search covers tickets, wiki, documents and feeds, not mail or the bank portal. The mailbox's default search leaves out spam and trash, and quoted text stays folded until expanded.

The firm's way

A playbook gives the route for each kind of charge. How a letter lists other charges, dates a chat or call, names the agent and names goods shows only in 71 past letters; no page states it.

People

None. The workspace accepts no writes, and both harnesses' ask‑the‑user tools were off.

Clock

None: a fixed Sunday evening. The harnesses also told their models the real date, Claude Code one before that moment and Codex one after.

Sandbox

A macOS sandbox profile let Claude Code read and write only its work folder; Codex ran read‑only, with a file tool confined to the output folders. Probes checked both before every attempt, and a marker file outside the folder never appeared.

Logs

The desk logs every call and the workspace every page served; neither is scored.

262 weighted checks in ten layers; leaving work out never pays

Structure

320 checks in the key, 58 of them repeats of another check, shown but not weighted: 262 count, in ten layers. A check that doesn't apply to an attempt drops out, so 191 to 262 counted in our attempts.

Layers

Tracker statuses and amounts 18%, tracker dates 14%, look‑alikes left alone 6%, the set of letters 10%, what each letter carries 9%, house style 12%, what letters keep out 7%, the note 16%, amounts that agree 5%, file form 3%.

House style

Three groups of 4%: the company's other charges listed, a chat or call dated and attributed as the firm does, goods named as the firm does. A group with nothing to score in an attempt drops out, its weight spread pro rata over everything scored.

No free passes

A missing letter fails its own check, its detail checks and its house‑style checks, so leaving a letter out never beats writing it with a slip. An empty attempt scores 0.

Who decides

Scripts decide 238 weighted checks. Two blind LLM readers, Opus 5.5 subagents, read the 23 note items on anonymized, shuffled notes; the task's coordinator reads one check on unsupported claims on the letters. The readers can run on a model you choose.

Reader agreement

On the 13 Anthropic attempts the readers split on 3 of 312 judgements, each settled by the coordinator's written ruling, recorded with the grades. The GPT attempts' notes were read later beside Anthropic notes as anchors; after four drifting items were re‑read, the readers agreed on all 276.

Grader self‑test

The reference answer passes all checks, and 25 near‑miss answers each fail exactly the checks aimed at them. Fairness gates, all passing, show every deciding record is reachable by reasonable searches, and that the obvious path alone doesn't reach the hidden ones.

Formula

Reward = the sum over layers of the layer's share × the fraction of its applying checks passed, divided by the shares that apply, from 0 to 1.

Attempts reached 94%, but house style cost every one and none fully solved it

With only the request, 21 of 24 attempts scored 77–94%; GPT‑6 Luna's three stopped after about a minute and scored 29–34%. None fully solved it. One model's three attempts lay 12 points apart (Sonnet 5.5, 79–91%).

Each dot is one attempt, three per model; the line marks the median.

Three attempts per model, with only the request; the table gives each model's mean. There is no version with a hint or the rules.

Not shown: Opus 5.5's one attempt with the same records as 23,553 plain files in a folder, without the workspace or the word counter, scored 94%. It found every hidden record and still lost marks on house style.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. A credited charge claimed again. 12 of 24 attempts asked a company for money its later bill had already credited; none of them had opened that bill. GPT‑6 Astra and GPT‑6.1 Sol, searching the mailbox company by company, caught it in all 6 of their attempts.

2. A wrong fee accepted. 8 of 24 attempts tracked a shop's fee as legitimate, though a record off the obvious path shows it was wrong; each had never opened that record or never expanded the quoted part that settles it.

3. Not opening, not misreading. Every wrong call on a fact off the obvious path came from a record never opened or a quoted part never expanded. GPT‑6 Astra and GPT‑6.1 Sol decided 35 of their 36 hidden facts right.

4. House style. It was the largest loss in each of the 21 attempts that wrote their letters, and no attempt wrote its letters fully the firm's way. GPT‑6 Astra and GPT‑6.1 Sol, best at finding hidden records, followed less of it than any Anthropic model; GPT‑6 Sol followed none.

5. Stopped early. GPT‑6 Luna ended each of its 3 attempts after about a minute, having written one or two of the twelve letters.

What an attempt costs

Length

With only the request, the median attempt took 12 minutes and 212 tool calls (19 to 432 across the 24), and about 17 million tokens, cached input included.

Cost

Recorded for 12 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the readers' calls aren't included.

What you license

With a licence

The workspace with its five data files and desk tools, the request and the plain‑folder control, the grader with its 23 note items and near‑miss answers, the fairness gates, the answer key and its reference answer, and the grader's report on each of the 25 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran on one Mac in each harness's sandbox, not in a container.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language. The workspace, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model and no told‑the‑rules version: the only control is one attempt on plain files, which can't separate access from the spread between attempts.
• One model's three attempts lay 12 points apart: with one attempt each, Sonnet 5.5 could have ranked anywhere from first to sixth of the eight.
• The note readers are Claude models (Opus 5.5), reading Claude and GPT notes; the anchors measure their drift, not a possible preference for Claude's writing.
• Readings were corrected for all attempts alike: two false fails removed during the Anthropic attempts, four readings after them and four after a fairness audit on 9 October; these eight moved single attempts up to 3.3 points and no model's place. The GPT attempts were graded after all of them.
• The request says its word counter counts as the grader does, so the agent is told it is graded.
• Two GPT attempts stalled when the machine went to sleep and were repeated from the start. Attempts ran on one Mac in each harness's sandbox, not a sealed container.
• Doing nothing scores 0 and the reference answer passes all checks (the grader's self‑test). No red‑team attempts yet.