Returns‑label request hiding unraised account problems
ID letter‑and‑record/task04 · revision 1 · tested 13 October 2026
At a home‑goods retailer's escalations desk, a customer asks only for a new returns label and says the rest is done. The agent has to give her the label, go through her whole account unasked and put right what nobody flagged.
- Checks
- 62, in eight weighted parts
- Tested
- 8 models, 3 attempts each
- Best attempt
- 98% (Fable 5.1)
- Median
- 94%
- Full solves
- 0 of 24
- Median attempt
- 4 min
She asks for a label; her account holds four problems she never raised
When to act or escalate
20% of the reward is for putting right, today, four problems she never raised, while goodwill she isn't owed fails.
Catching errors
The problems sit in her account unflagged, and a colleague's unsent draft puts the label against the wrong return.
Disclosure and honesty
The reply must name no colleague and no other customer, and must not state a spending limit.
Unstated requirements
Nothing asks the agent to look past the label; two closed cases show the team going through a whole account.
A read‑only hub, 65 files and a colleague's unsent draft
The agent's world
A text browser into the company's read‑only service hub, through the model's usual agent, with no web.
What's in it
65 files, including ten closed cases, her messages and a colleague's draft reply, partly right; search reads titles, not contents.
Nobody to ask
Nobody answers, and every attempt sees the same hub at a fixed Monday‑morning moment.
The reward
62 checks in eight fixed‑weight parts; the four unraised problems alone are worth 20%, and scripts decide all but nine.
Two‑thirds of attempts scored 90–98%, and the rest 51–79%
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Fable 5.1Anthropic | 98% |
| Opus 5Anthropic | 96% |
| GPT‑6.1 SolOpenAI | 96% |
| Opus 5.5Anthropic | 94% |
| GPT‑6 AstraOpenAI | 94% |
| Sonnet 5.5Anthropic | 82% |
| Sonnet 5Anthropic | 72% |
| GPT‑6 LunaOpenAI | 52% |
With the three house‑style rules spelled out, 11 of the 24 attempts fully solved it.
They search too narrowly, slip on the house style, and leave money unpaid
Catching errors
29% of attempts missed one problem by searching the payment records too narrowly or not at all, each losing 19 points.
When to act or escalate
Three models put all four unraised problems right in only one of their nine attempts; the other five did in every attempt.
In 13% of attempts, nothing she was owed was paid that day: all three of GPT‑6 Luna's.
Unstated requirements
96% of replies named an item differently from past replies, including nearly all of the five strongest models'.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 1.6 million tokens, cached input included.
What's in it
The hub with its 65 files, the request in three versions up to the house‑style rules, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the hub runs as a local server, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials