Onboarding bill review with a simulated client
ID letter‑and‑record/task05 · revision 1 · tested 14 and 19 October 2026
At a household bill‑management firm, the agent does a new client's first bill review, writing letters, a tracker and a note. Three charges turn on facts only she has; she can be messaged, though nothing in the request or case ticket says so.
- Checks
- 295–320, in ten weighted parts
- Tested
- 8 models, 3 attempts each
- Best attempt
- 84% (GPT‑6 Astra)
- Median
- 78%
- Full solves
- 0 of 24
- Median attempt
- 10 min
A new client's first review, where three answers are only with her
Clarifying with people
Three charges turn on facts only the client has; she answers only the right question, and nothing says she can be messaged.
Multimodal records
Facts sit across tickets, a wiki, client uploads, a mailbox and a bank portal: about 90,000 records searched through a text browser.
Catching errors
Statement names differ from shop names, companies were renamed, and refund messages must be checked against the account.
Disclosure and honesty
Each letter is read for claims that neither a record nor the client's answers support.
When to act or escalate
Disputing what the client could have explained, or writing letters where none is due, fails checks; promised refunds must be held as pending.
Unstated requirements
Six habits, in the letters, tracker and note, show only in the firm's latest past letters.
A workspace of 90,000 records and a client who answers 20 messages
The agent's world
A text browser into the firm's read‑only workspace; the agent writes files to an output folder, and its only form is the client's message box.
The client
She answers at once from 25 fact cards, worded by an AI model, and volunteers nothing; she stops after 20 messages.
Same question, same answer
Code matches questions to facts, with one cached AI call for wordings it can't match; saved replies keep each exchange the same.
The reward
In ten fixed‑weight parts, so a varying check count doesn't shift the weights; asking her isn't scored, only what the attempt writes.
88% of attempts scored 69–84%, and the other three fell far below
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| GPT‑6 AstraOpenAI | 82% |
| Fable 5.1Anthropic | 80% |
| GPT‑6.1 SolOpenAI | 80% |
| Opus 5.5Anthropic | 79% |
| Opus 5Anthropic | 79% |
| GPT‑6 SolOpenAI | 73% |
| Sonnet 5.5Anthropic | 70% |
| GPT‑6 LunaOpenAI | 26% |
In five extra attempts with one sentence added saying she can be messaged, every attempt asked her and settled all three charges only she could explain.
They decide for the client, claim in her name, and miss charges under other names
Clarifying with people
Every attempt opened her thread, and none asked her anything; 79% disputed a card charge only she could explain.
Disclosure and honesty
25% of attempts wrote a dispute letter stating, as her words, that no one else had used her card, though she was never asked.
Catching errors
Every attempt by Opus 5.5, Fable 5.1 and Sonnet 5.5 missed a monthly charge billed under another company's name.
Unstated requirements
House style was the largest loss in every attempt that wrote its letters.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 10 million tokens, cached input included.
What's in it
The workspace, the client's service and fact cards, the request in three versions, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the workspace and the client run as local services, and the client's voice can use any model.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials