Retail escalation reply and case records
ID letter‑and‑record/task01 · revision 21 · tested 1 and 21 October 2026
Covering a home‑goods retailer's escalations desk, the agent answers a complaint that has waited since Saturday and writes up the case, working only from the company's read‑only service hub, with nobody to ask.
- Checks
- 114–115, in seven weighted parts
- Tested
- 15 models, one attempt each
- Best
- 98% (Opus 5.5)
- Median
- 73%
- Full solves
- 0 of 15
- Median attempt
- 6 min, 102 tool calls
Answer an overdue complaint, and write up the case the team's way
Unstated requirements
The team's habits for naming items, orders, earlier refunds and tax splits show only in the replies and notes of eight closed cases.
When to act or escalate
Each amount she raises must be paid today, referred to the manager, routed to another team or declined, and paying what needs referral fails.
Disclosure and honesty
18% of the reward is for what the reply keeps out, such as internal flags and other customers, if it still tells her enough.
Catching errors
The case file holds misfiled documents, three policy versions and a colleague's draft with wrong figures, so each source must be weighed.
A read‑only service hub, 67 files and nobody to ask
The agent's world
A text browser into the company's read‑only service hub, through the model's usual agent, with no web.
What's in it
67 files: eight closed cases, three policy versions, a colleague's unsent draft and other customers' records; search reads titles, not contents.
Nobody to ask
Nobody answers, and every attempt sees the same hub at a fixed Monday‑morning moment.
The reward
114–115 checks on the four files, in seven fixed‑weight parts; scripts decide 68 of them and two AI judges the rest.
The five strongest scored 87–98%, and seven fell below 40%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 98% |
| Fable 5.1Anthropic | 97% |
| Opus 5Anthropic | 95% |
| GPT‑6.1 SolOpenAI | 90% |
| GPT‑6 AstraOpenAI | 87% |
| Sonnet 5Anthropic | 75% |
| Sonnet 5.5Anthropic | 74% |
| GPT‑6 SolOpenAI | 73% |
| GPT‑6 LunaOpenAI | 37% |
| GLM‑5Zhipu | 30% |
| Kimi K2.5Moonshot | 24% |
| GPT‑5.6 LunaOpenAI | 16% |
| Qwen3‑235BAlibaba | 12% |
| DeepSeek V3.2DeepSeek | 11% |
| Nemotron Super 3NVIDIA | 10% |
With every rule the team works to spelled out in the request, two of the fifteen fully solved it and eight scored 91% or more.
They miss the team's habits, hold back a refund that was due, and tell her too little
Unstated requirements
No reply described the re‑issued order with the details past replies give, and 87% of notes left out the goods‑and‑tax split that past notes show for earlier refunds.
80% of notes didn't record which of two conflicting instructions they followed, with both dates.
When to act or escalate
A refund the card issuer had sent back was due again that day, and 67% of attempts didn't pay it.
Disclosure and honesty
In 47% of attempts, the reply told her too little to earn any of the reward for what it kept out.
What an attempt takes, and what the environment includes
An attempt
The median attempt used about 2.8 million tokens, cached input included.
What's in it
The hub with its 67 files, the request in three versions up to every rule spelled out, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; the hub runs as a local server, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials