Retail escalation reply and case records

An overdue complaint, decided and written up from the records alone

Role

Covering an absent colleague's escalations queue at a home‑goods retailer on a Monday morning.

Request

A customer's complaint arrived on Saturday and is still unanswered. Answer her and write up the case, with its money and timeline records, as four named files, from the hub alone, since nobody is available.

Deliverable

Four files: a reply email under 600 words, a case note, a money sheet with one row per amount and a timeline with one row per dated event, with the columns the record format sets.

What the reward depends on

Deciding each amount she raises (pay today, refer to the manager, route to another team or decline) under the policy in force; what the reply tells her and keeps out; a note that records both sides and the company's own failures; the house style.

Where the requirements are

Mostly unstated. They show in eight closed cases, the manager's messages and which of three policy versions governs. Some facts sit where no link from the case leads, and a recently closed look‑alike case points the wrong way.

A read‑only service hub of 67 files, reached one text page at a time

Interface

A text browser into the company's service hub, served over MCP and driven by each model's own agent harness: Claude Code, Codex CLI, Kimi CLI, Qwen Code or Prime Agent in our attempts. No shell, no web, and no case files in the workspace.

Desk tools

Eight: open a page by its path, click a numbered link, read, scroll, back, type and submit a search or filter, and a calculator. A screen is a page's title, path and up to 60 lines of text.

The hub

Records built from the files: 7 accounts, 8 contacts, 17 cases, 9 orders, 3 returns, a call, a carrier claim, a knowledge base, a refunds report, the team mailbox and the team chat. Each message, note and policy reads as its file's own page.

Files

67, filed in 15 folders: 26 text, 18 emails, 7 CSV (one empty), 6 PDF, 6 Word, 3 spreadsheets and a zip of 3. Among them: eight closed cases, three policy versions, a colleague's unsent draft and six other customers' records.

Finding things

No page lists every file, and file pages have hashed ids. Search matches record fields and file titles, never contents, up to 50 results; lists show 10 rows a page, spreadsheets 25 and the team chat 15 lines.

People

None. Nobody answers, and there is no tool to send a message.

Clock

None. Every page shows the same Monday‑morning desk time, and the hub refuses every change, so every attempt sees the same hub.

Sandbox

The first 30 attempts ran in a sealed Docker container that reached only the model proxy and the attempt's own desk server. The 15 added on 21 October ran on one Mac in a sandbox profile, with the same tools.

Logs

The hub logs every page served and the desk every call, kept with each attempt to check its measurement. The grader reads only the four files.

115 checks in seven fixed layers; what the reply leaves out pays only if it says enough

Structure

115 yes/no checks in seven layers, each a fixed share: files and form 4%, facts in the reply 14%, what it tells her 20%, what it keeps out 18%, the case note 28%, the money sheet 8%, the timeline 8%.

Gate

The 13 keep‑out checks pay only if the reply passes at least 9 of the 14 on what it tells her and the reply and note reach 60 and 150 words; seven also need their twin in the note. A short, clean reply earns nothing for what it left out.

Checks that apply

One timeline row counts only when the note gives its date, so an attempt is scored on 114 or 115 checks; one refund's goods‑and‑tax split is required only when the note states that refund.

Who decides

Scripts decide 68 checks: addresses, amounts, dates, identifiers and every data‑sheet row, each figure, date and identifier matched as a whole word. Two LLM judges decide 47, one yes/no question per check with numbered clauses; both must agree. Either judge can run on a model you choose.

Judge models

Opus 5.5 and GPT‑5.6 Sol, on Bedrock at temperature 0, for the first 30 attempts; Opus 5.5 and GPT‑6.1 Sol for the 15 added, each first checked against the stored grades.

Judge agreement

The judges split on 31 of 1,849 judged results across the 45 attempts (1.7%). The environment's author ruled each, the first 20 after an independent reviewer had ruled them blind; one agreed verdict was corrected after an audit.

Grader self‑test

The reference answer passes all 115 checks, with both judges agreeing; an empty folder scores 0. Of 57 near‑miss answers, each wrong in one known way, Opus 5.5 failed all 57 on the check aimed at them, GPT‑5.6 Sol 56.

Formula

Reward = the sum over layers of the layer's share × the fraction of its applying checks passed, from 0 to 1.

Five models scored 87–98%; none fully solved it with only the request

With only the request, the 15 attempts, one per model, scored 10–98%: the five strongest 87–98%, and the seven that opened at most 30 of the 67 files 10–37%. Told the rules, 2 of 15 fully solved it and 8 scored 91% or more.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus a paragraph saying the company's expectations show up as patterns across its documents, with three worked examples from an unrelated company.

Told the rules: the request plus the team's 34 rules in five groups, with no figures, dates or identifiers, and no facts from the files.

What the models got wrong

Counted over the 15 models, one attempt each, with only the request, unless a line says otherwise.

1. Habits shown only in past cases. All 15 replies named a re‑issued order without the details past replies give, and 13 of 15 notes gave earlier refunds as one figure where past notes split goods and tax. Told the rules, the eight strongest models missed neither.

2. A returned refund held back. 10 of 15 attempts didn't re‑issue that day a refund the card issuer had sent back. GPT‑6.1 Sol, GPT‑6 Astra and GPT‑6 Sol held it for a trace, though the policy and past cases they had read re‑issue it; all five Anthropic models paid it.

3. Conflicting instructions, unrecorded. 12 of 15 notes didn't record which of two conflicting instructions they followed, with both dates; Opus 5.5, Fable 5.1 and Opus 5 did.

4. Read little, filled the gaps. The seven lowest each opened at most 30 of the 67 files, against 34 to 55 for the rest. None said enough to earn any of the 18% for what the reply kept out, and two paid everything she asked for.

5. Help didn't close the gap. Told the rules, eight models scored 91–100%, but the other seven scored 0–79%: two saved nothing, and GPT‑6 Luna stopped after 21 seconds with a reply holding only a subject line.

What an attempt costs

Length

With only the request, the median attempt took 6 minutes and 102 tool calls (4 to 201 across the 15), and about 2.8 million tokens, cached input included (14 of the 15 logged tokens).

Cost

Not recorded for most attempts: 6 of the 15 with only the request carry one, five as Claude Code's own figure at list prices and one estimated from logged tokens, so we give no median. That's the agent model only; the judges' calls aren't included.

What you license

With a licence

The hub with its 67 files and desk tools, the request in all three versions, the grader with its 47 judge questions and near‑miss answers, the answer key and its reference answer, and the grader's report on each of the 45 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran in a sealed Docker container or, for the 15 added, a sandboxed Mac, outside Harbor.

Revisions

Revision 21 is the one tested and offered. Its answer key was corrected after an audit on 9 October and a review of the added attempts on 22 October; all attempts were regraded each time.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• Each model ran in one harness, not always its maker's, so a score is for the pair. Prime Agent's two models reached the hub through a Python kernel that could loop over pages; the other harnesses couldn't script it.
• 47 of the 115 checks depend on LLM judges; their 31 splits across the 45 attempts were settled by the author's written ruling.
• The 15 attempts added on 21 October ran on one Mac, on newer harness releases and with a different second judge; three stored attempts replayed through the local hub returned 533 of 533 screens word for word.
• An audit on 9 October found seven grading faults, each worth 0.3 to 1.4 points to an attempt. All were fixed and every attempt regraded, so the results here include the fixes.
• Nine first attempts by three open‑weight models were voided by faults in our desk server and repeated after the fix; the scores are from the repeats.
• Doing nothing scores 0 and the reference answer 1 (the grader's self‑test). No red‑team attempts yet.