Missing‑payments reply from error‑laden retail records
ID letter‑and‑record/task02 · revision 1 · tested 12–13 October 2026
At a home‑goods retailer's escalations desk, a customer says four payments never reached her card. The agent finds out what happened to each, pays what is owed and writes up the case, from records that carry filing errors.
- Checks
- 61, in eight weighted parts
- Tested
- 8 models, 3 attempts each
- Best attempt
- 99% (Opus 5.5)
- Median
- 78%
- Full solves
- 0 of 24
- Median attempt
- 5 min
Four payments she says never arrived, and records with filing errors
Catching errors
Each payment she chases turns on a different record error or a claim the records don't support, so an empty search proves nothing.
Unstated requirements
Four habits for naming items, orders, earlier refunds and tax splits show only in ten closed cases.
Disclosure and honesty
The reply may not reveal another customer's account, name colleagues or state spending authority, while still owning the company's mistakes.
A read‑only hub, 66 files and ten closed cases
The agent's world
A text browser into the company's read‑only service hub, through the model's usual agent, with no web.
What's in it
66 files, including ten closed cases and a refund ledger with other customers' rows; search reads titles and record fields, not contents.
Nobody to ask
Nobody answers, and every attempt sees the same hub at a fixed Monday‑morning moment.
The reward
61 checks in eight fixed‑weight parts; one hard‑to‑find payment alone is worth 20%, and scripts decide all but nine.
Attempts ranged from 37% to 99%, largely on one payment worth 20%
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Opus 5Anthropic | 93% |
| Opus 5.5Anthropic | 91% |
| GPT‑6.1 SolOpenAI | 85% |
| Fable 5.1Anthropic | 84% |
| GPT‑6 AstraOpenAI | 83% |
| Sonnet 5Anthropic | 72% |
| Sonnet 5.5Anthropic | 67% |
| GPT‑6 LunaOpenAI | 46% |
With the four house‑style rules spelled out, two of the 24 attempts fully solved it, but a third still missed the payment worth 20%.
They take a search for the record, miss what settles a payment, and slip on the house style
Catching errors
42% of attempts misread the records on the payment worth 20% and wrote it up untested, many right after reading a past case that pointed the right way.
33% of attempts missed the record that settles a third payment, some taking an earlier write‑up's word for it.
Unstated requirements
92% of attempts wrote an item, order, return or charge differently from past replies and notes.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 2.1 million tokens, cached input included.
What's in it
The hub with its 66 files, the request in three versions up to the house‑style rules, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the hub runs as a local server, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials