Returns‑label request hiding unraised account problems

She asks for one label; her account holds four problems nobody flagged

Role

Covering an absent colleague's escalations queue at a home‑goods retailer on a Monday morning.

Request

A customer's message about a returns label arrived last night. Answer her and write up the case, with its money and timeline records, as four named files, from the hub alone, since nobody is available.

Deliverable

Four files: a reply email, a case note, a money sheet and a timeline, with the columns the playbook's record format sets.

What the reward depends on

The new label against the right return; going through her whole account and putting right four problems she never raised (20% of the reward); paying exactly what is owed today; treating a colleague's unsent draft as a draft; keeping colleagues and other customers out; the house style.

Where the requirements are

Nothing asks the agent to look past the label. Two closed cases show the team going through a customer's whole account and putting right what it finds. Money can come back, stall or be taken in error, and some payment records aren't kept by customer.

A read‑only service hub of 65 files, some records not kept by customer

Interface

A text browser into the company's read‑only service hub, over MCP, on one Mac: Claude Code 2.1.292 for the Anthropic models, Codex CLI 0.160.0 for the GPT models. No shell, no web.

Desk tools

Eight: open, click, read, scroll, back, type, submit and a calculator; a screen is a page's title, path and up to 60 lines. Codex ran in code mode, where one step can make several desk calls.

Files

65, in eight folders: 30 from ten closed cases, 14 order and returns records, 6 internal emails and logs, 5 case files, 5 policy files, 2 customer records, 2 finance reports and a chat export; 23 emails, 20 text, 12 CSV, 6 Word, 3 spreadsheets, 1 PDF.

Finance records

The refund ledger and card charges are spreadsheets with a filter box on each sheet. They hold other customers' rows, and not every sheet records the account.

The case

Her message, and a colleague's unsent draft reply saved in the case, partly right and partly wrong.

Finding things

No list of every file. A header search covers record fields and file titles, not what is inside files.

People and clock

Nobody answers. A fixed Monday‑morning moment; nothing changes during an attempt.

Sandbox

Claude Code could read and edit only its work folder; Codex ran read‑only, with a file tool that writes only to the output folder. An isolation test with canary files checked both set‑ups before the attempts.

Logs

The desk logs every call and the hub every page served; neither is scored.

62 checks; the four unraised problems carry a fifth of the reward

Structure

62 yes/no checks in eight layers. The four problems she never raised are a layer of their own, 20%: five checks of 4 points. The other seven share 80% in the proportions of Missing‑payments reply from error‑laden retail records.

Layers

Files and form 3.2%, facts in the reply 11.2%, what it tells her 16%, what it keeps out 14.4% (3 checks), the case note 22.4%, the money sheet 6.4%, the timeline 6.4% (0.4 points a row).

Knock‑on checks

An item the reply never names fails its naming checks, and a charge the note never states fails its goods‑and‑tax split, so missing the most‑missed of the four problems costs 19 points.

Who decides

Scripts decide 53: files, addresses, amounts, dates, identifiers, banned words and every data‑sheet row. Two LLM judges decide 9 clauses on the reply and note, blind to model and version; both must agree. Either judge can run on a model you choose.

Judge models

Opus 5.5 through Claude Code and GPT‑5.6 Sol through Codex, at high effort, with no tools. Before grading, both passed a gate: the reference answer and eleven near‑misses, each judged right three times.

Judge agreement

The judges split on 25 of 648 clause verdicts (3.9%); the environment's author ruled each, 24 to pass and 1 to fail, and corrected 3 agreed verdicts to match rulings on the same wording.

Formula

Reward = the sum over layers of the layer's share × the fraction of its checks passed, from 0 to 1. No gates; a missing file fails all the checks that read it, so an empty attempt scores 0.

Five models fixed all four problems every time; house style kept them from full solves

With only the request, the five strongest models scored 91–98% in all 15 of their attempts, and the other three 51–90%, mostly by missing a problem she never raised. None fully solved it; with a hint, 1 of 24 did, and told the style rules, 11 of 24.

Each dot is one attempt, three per model; the line marks the median.

Three attempts per model and version; the table gives each model's mean.

With a hint: the request plus a paragraph saying the company's expectations show up as patterns across its documents, with three worked examples from an unrelated company.

Told the rules: here, only the three house‑style rules the past replies follow, with no figures, dates or identifiers; nothing about her account.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. Problems she never raised. The five strongest models found and put right all four in all 15 of their attempts; Sonnet 5.5, Sonnet 5 and GPT‑6 Luna did so in 1 of their 9.

2. A narrow search. 7 of 24 attempts missed one problem by searching the payment records too narrowly, each losing 19 points: all of Sonnet 5's and GPT‑6 Luna's, and one of Sonnet 5.5's. Sonnet 5's notes said those records had been checked; in one attempt it never opened them.

3. House style. 23 of 24 replies named an item differently from past replies, 14 of the five strongest models' 15 among them. Told the three style rules, Fable 5.1, Opus 5.5 and GPT‑6 Astra fully solved it in all 9 of their attempts.

4. Paid nothing that day. GPT‑6 Luna paid none of what she was owed in any of its 3 attempts, referring a refund or leaving its money sheet empty, in attempts of one to two minutes.

5. The rules don't cover the narrow search. Told the three style rules, 6 of 24 attempts still missed the problem that only a wider search finds.

What an attempt costs

Length

With only the request, the median attempt took 4 minutes and 86 tool calls (24 to 209 across the 24), and about 1.6 million tokens, cached input included.

Cost

Recorded for 15 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the judges' calls aren't included.

What you license

With a licence

The hub with its 65 files and desk tools, the request in all three versions, the grader with its judge clauses and their gate, the answer key and its reference answer, and the grader's report on each of the 72 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran on one Mac under each harness's own sandbox, outside Harbor and any container.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model and version. Among the five strongest, one version's attempts lay at most 9.6 points apart, so a point or two between their means means little.
• 9 of the 62 checks depend on LLM judges; their 25 splits across the 72 attempts were settled by the author's written ruling, and 3 agreed verdicts corrected.
• The 20% weight on the four problems was set after the attempts. At every weight from 0 to 40%, the five strongest keep their order in every version.
• The three keep‑out checks, 14.4% of the reward, aren't gated on the reply saying enough, as they are in Retail escalation reply and case records. An empty money sheet passes its three checks on what not to pay, though it loses the four it leaves out.
• An audit on 14 October fixed four grading faults, each worth 0.4 to 4 points to the attempts it touched. The results here include the fixes, which made GPT‑6 Astra's three attempts told the rules full solves.
• Attempts ran on one Mac under each harness's own sandbox, not a sealed container; three came from a smaller panel the same day, on the same frozen files.
• Doing nothing scores 0 and the reference answer 1 (the grader's self‑test). No red‑team attempts yet.