Onboarding bill review with a simulated client

A new client's first review, where three charges turn on what only she knows

Role

The review desk of a household bill‑management firm on a Sunday evening, covering an absent account manager: a new client's onboarding review.

Request

Work one ticket: write any letters, a tracker and a note to the client as files, in the formats given, the note at most 2,200 words. Nothing says the client can be messaged.

Deliverable

One letter per company still owing her money (card disputes to her bank, claims to companies), a tracker CSV with fixed columns and statuses, and a note.

What the reward depends on

Three charges that turn on facts only she has; charges filed under other names or renamed companies; refunds and waivers that only look settled; look‑alikes left alone; letters that claim nothing no record or answer supports; six house habits, 16%.

Where the requirements are

The playbook gives the routes and lists her messages among what she has given the firm. The firm's latest letters and review packs show the habits. The three deciding facts are in no record, only with her.

90,151 records and one simulated client who answers only what she's asked

Interface

The text browser and nine desk tools of Household bill review: card disputes, claims, tracker, on one Mac: Claude Code 2.1.292 for the Anthropic models, Codex CLI 0.162.1 for the GPT models. The workspace's only form that changes anything is the send box on a client's message thread.

Scale

90,151 records for 300 households: 7,257 tickets and 686 wiki pages, 75 of them past letters and review packs, and for her 14 files, a mailbox of 6,714 messages, 834 card transactions, card statements and a checking feed.

Finding things

No list of records. The header search covers tickets, wiki, documents and feeds, not mail, the bank portal or messages; quoted text in emails stays folded until expanded.

Simulated client

One client, reached by typing into her message thread and sending. She answers in the same page load, from 25 fact cards and 4 fact‑free default lines. The six other clients the desk holds answer one fixed line with no facts.

Fact release

Code releases cards by routes on topic, company, card descriptor, amount, goods and dates, with one cached classifier call for a paraphrase no route matches. Every question with a card is answered; nothing is volunteered.

Voice

Sonnet 5.5 words the released lines. A reply that adds, drops or changes a fact, or gives advice, is sent back once, then replaced by the cards' own text. A packaged version can use a model you choose.

Guards

Catch‑all and injection‑style questions release no card, and a message naming no item gets a fact‑free line. She answers 20 messages an attempt, saying it's late from the 18th, then only that she's off to bed.

Same question, same answer

Replies are saved by a hash of the exchange, so the same exchange gets the same reply in every attempt.

Clock

A fixed Sunday evening. Only the time stamped on a sent message moves, with the attempt's elapsed time, capped at 9:59 pm.

Sandbox

Each harness in its sandbox, as in the monthly bill review. The client's service runs outside them all, and probes checked that the agents couldn't reach her cards, the key or the grader.

322 weighted checks; asking her isn't scored, only what the agent then writes

Structure

324 checks in the key; 2 note items are shown but not weighted, so 322 count, in ten layers. A check that doesn't apply drops out: 295 to 320 counted in our attempts with only the request.

Layers

Tracker statuses 17.2%, dates 13.4%, look‑alikes 5.7%, the set of letters 9.5%, letter contents 8.6%, house style 16% (four groups of 4), keep‑out 6.7%, the note 15.3%, amounts that agree 4.8%, file form 2.8%; no group of related checks above 8 points.

Messages

Whether an attempt messages the client isn't scored. Her answers count only through what the attempt then writes; the three charges only she can settle cost each of our 24 attempts with only the request 3.9 to 6.3 points.

Who decides

Scripts decide 296. Two blind LLM readers, Opus 5.5 subagents, read 27 note items, 25 of them weighted, with three reference notes of 81 known verdicts in every batch. The task's coordinator reads one check on unsupported claims on the letters.

Reader agreement

The readers matched all 81 known verdicts in each of four batches and split on 8 of 675 judgements (1.2%), each settled by the coordinator's written ruling, all to a pass.

Rulings

Every lost check was read against the records before scoring. Rulings widened what passes, for all attempts alike, except one tightening the author chose after seeing it cost six attempts 1.4 points each.

Grader self‑test

The reference answer passes all 324 checks, 35 near‑miss answers each fail exactly the checks aimed at them, and an empty attempt scores 0. Fairness gates, all passing, show every deciding record and card is reachable, asking her included.

Formula

Reward = the sum over layers of the layer's share × the fraction of its applying checks passed, divided by the shares that apply, from 0 to 1.

No attempt asked the client; told it could, every attempt did

With only the request, 21 of 24 attempts scored 69–84% and GPT‑6 Luna's three 1–45%; none fully solved it. None asked the client anything, so every attempt lost 3.9 to 6.3 points on the three charges only she could settle.

Each dot is one attempt; the line marks the median.

Three attempts per model with only the request; one told the rules, by Opus 5.5.

Told the rules: the request plus a paragraph naming every rule, the six house habits and what to ask the client about, never her answers.

Not shown: one added sentence saying she can be messaged, tested on 19 October. All 5 attempts (Opus 5.5 once, GPT‑6.1 Sol and GPT‑6 Astra twice) asked her and settled all three charges, scoring 85–89%.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Deciding for the client. None asked her, though all 24 opened her thread: 19 disputed a card charge only she could explain, 18 wrote that a parcel never came, and 20 never took up a return she had sent. Told she can be messaged, all 5 attempts asked.

2. Claims in her name. 6 of 24 attempts wrote a dispute letter stating, as her own words, that no one else had used her card, which nobody had asked her; the 6 attempts that asked her all passed that check.

3. Charges under other names. All 9 attempts of Opus 5.5, Fable 5.1 and Sonnet 5.5 missed a monthly charge billed under another company's name. GPT‑6 Astra and GPT‑6.1 Sol lost under 1.5 points an attempt on the eight charges hidden behind other names, settled‑looking records or recalculation.

4. House style. The largest loss in each of the 21 attempts that wrote their letters: a mean of 13.1 to 15.2 points an attempt, of the 16 it carries, for GPT‑6 Astra, GPT‑6.1 Sol, GPT‑6 Sol and Sonnet 5.5, and 6.7 to 9.7 for the other Anthropic models.

5. A return missed, even told the rules. Opus 5.5's one attempt told the rules asked her once, settled two of the three charges and scored 95%, but never took up the return she had sent.

What an attempt costs

Length

With only the request, the median attempt took 10 minutes and about 161 tool calls (26 to 344 across the 24), and about 10 million tokens, cached input included.

Cost

Recorded for 12 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the client's voice and readers aren't included.

What you license

With a licence

The workspace with its five data files and desk tools, the client's service and fact cards, the request in all three versions, the grader with its readers' instruction, the fairness gates, the answer key and its reference answer, and the grader's report on each of the 30 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran on one Mac in each harness's sandbox, not in a container.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language. The workspace, fact cards, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model with only the request; the told‑the‑rules version was tested once and the permission sentence 5 times, too few to order models on them.
• The GPT models' harness tells them not to message others unless told to, and the request doesn't. The three client‑only charges stay in the reward; left out, each model's mean rises 3.9 to 6.4 points (GPT‑6 Luna's 0.9), and the order holds.
• Asking isn't scored, only what an attempt then writes, so a good question earns nothing by itself.
• One documented gap: disputing two correct bills would take two house‑style instances out of scoring. No attempt did so.
• The client's voice was Sonnet 5.5 in every attempt. Code releases her facts, but we haven't measured the reward under a second voice model.
• The note readers are Claude models (Opus 5.5), reading Claude and GPT notes; the reference group measures their drift, not a possible preference for Claude's writing.
• Pilot attempts by four models are on the board. The environment was built to keep its top three at or below 80%; only one of its top three, GPT‑6.1 Sol, is. An audit on 15 October widened four readings for all attempts, moving model means by -0.1 to +1.8 points.
• Doing nothing scores 0 and the reference answer passes all checks (the grader's self‑test). No red‑team attempts yet.