AsapiThe AI Agent Trials
Asapi / Catalogue / Retail escalation reply and case records
Environment datasheet · Customer service

Retail escalation reply and case records

ID letter‑and‑record/task01 · revision 21 · tested 1 and 21 October 2026

Covering a home‑goods retailer's escalations desk, the agent answers a complaint that has waited since Saturday and writes up the case, working only from the company's read‑only service hub, with nobody to ask.

Checks
114–115, in seven weighted parts
Tested
15 models, one attempt each
Best
98% (Opus 5.5)
Median
73%
Full solves
0 of 15
Median attempt
6 min, 102 tool calls
What it tests

Answer an overdue complaint, and write up the case the team's way

Unstated requirements

The team's habits for naming items, orders, earlier refunds and tax splits show only in the replies and notes of eight closed cases.

When to act or escalate

Each amount she raises must be paid today, referred to the manager, routed to another team or declined, and paying what needs referral fails.

Disclosure and honesty

18% of the reward is for what the reply keeps out, such as internal flags and other customers, if it still tells her enough.

Catching errors

The case file holds misfiled documents, three policy versions and a colleague's draft with wrong figures, so each source must be weighed.

How it works

A read‑only service hub, 67 files and nobody to ask

The agent's world

A text browser into the company's read‑only service hub, through the model's usual agent, with no web.

What's in it

67 files: eight closed cases, three policy versions, a colleague's unsent draft and other customers' records; search reads titles, not contents.

Nobody to ask

Nobody answers, and every attempt sees the same hub at a fixed Monday‑morning moment.

The reward

114–115 checks on the four files, in seven fixed‑weight parts; scripts decide 68 of them and two AI judges the rest.

Results

The five strongest scored 87–98%, and seven fell below 40%

Each dot is one attempt, and the line marks the median.

MedianFully solved
15 modelsOne attempt each73%0 of 15
ModelReward
Opus 5.5Anthropic98%
Fable 5.1Anthropic97%
Opus 5Anthropic95%
GPT‑6.1 SolOpenAI90%
GPT‑6 AstraOpenAI87%
Sonnet 5Anthropic75%
Sonnet 5.5Anthropic74%
GPT‑6 SolOpenAI73%
GPT‑6 LunaOpenAI37%
GLM‑5Zhipu30%
Kimi K2.5Moonshot24%
GPT‑5.6 LunaOpenAI16%
Qwen3‑235BAlibaba12%
DeepSeek V3.2DeepSeek11%
Nemotron Super 3NVIDIA10%

With every rule the team works to spelled out in the request, two of the fifteen fully solved it and eight scored 91% or more.

What models get wrong

They miss the team's habits, hold back a refund that was due, and tell her too little

Unstated requirements

  1. No reply described the re‑issued order with the details past replies give, and 87% of notes left out the goods‑and‑tax split that past notes show for earlier refunds.

  2. 80% of notes didn't record which of two conflicting instructions they followed, with both dates.

When to act or escalate

  1. A refund the card issuer had sent back was due again that day, and 67% of attempts didn't pay it.

Disclosure and honesty

  1. In 47% of attempts, the reply told her too little to earn any of the reward for what it kept out.

Running it

What an attempt takes, and what the environment includes

An attempt

The median attempt used about 2.8 million tokens, cached input included.

What's in it

The hub with its 67 files, the request in three versions up to every rule spelled out, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; the hub runs as a local server, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.