Missing‑payments reply from error‑laden retail records

Four payments she says never arrived, traced through records with filing errors

Role

Covering an absent colleague's escalations queue at a home‑goods retailer on a Monday morning.

Request

A customer's complaint arrived on Saturday and is still unanswered. Answer her and write up the case, with its money and timeline records, as four named files, from the hub alone, since nobody is available.

Deliverable

Four files: a reply email, a case note, a money sheet and a timeline, with the columns the playbook's record format sets.

What the reward depends on

Finding out what happened to each payment she chases; paying what is owed today, once, as one total, and nothing more; declining what isn't owed, with its basis; owning the company's mistakes; keeping other customers and colleagues out; the house style.

Where the requirements are

In the records, where a payment can be filed against the wrong record, so an empty search proves nothing and a payment on file may not be what it seems; in ten closed cases, two of which look like hers and lead to different write‑ups; and in the manager's emails.

A read‑only service hub of 66 files, with filtered sheets and nobody to ask

Interface

A text browser into the company's read‑only service hub, over MCP, on one Mac: Claude Code 2.1.292 for the Anthropic models, Codex CLI 0.160.0 for the GPT models. No shell, no web.

Desk tools

Eight: open, click, read, scroll, back, type, submit and a calculator; a screen is a page's title, path and up to 60 lines. Codex ran in code mode, where one step can make several desk calls.

Files

66, in eight folders: 30 from ten closed cases, 12 order and returns records, 7 case files, 6 internal emails and logs, 5 policy files, 3 customer records, 2 finance reports and a chat export; 24 text, 20 emails, 12 CSV, 6 Word, 4 spreadsheets.

Finance records

The refund ledger and card payments are spreadsheets shown sheet by sheet, with a filter box that keeps the rows containing what is typed. They hold other customers' rows too.

Finding things

No list of every file. A header search covers record fields and file titles, not what is inside files.

People

None. A colleague who handled several of the closed cases is away too, and nobody answers.

Clock

None: a fixed Monday‑morning moment, and nothing changes during an attempt.

Sandbox

Claude Code could read and edit only its work folder; Codex ran read‑only, with a file tool that writes only to the output folder. An isolation test with canary files checked both set‑ups before the attempts.

Logs

The desk logs every call and the hub every page served; neither is scored.

61 checks; one payment carries a fifth of the reward

Structure

61 yes/no checks in eight layers with fixed weights. One payment whose record is hard to find is a layer of its own, 20%: two checks of 10 points. The other seven layers share 80%.

Layers

Files and form 3.2%, facts in the reply 11.2%, what it tells her 16%, what it keeps out 14.4% (3 checks), the case note 22.4%, the money sheet 6.4%, the timeline 6.4% (about 0.5 points a row).

No gates

No check is make‑or‑break and none is gated. A missing file fails all the checks that read it, so an empty attempt scores 0.

Who decides

Scripts decide 52: files, addresses, amounts, dates, identifiers, banned words and every data‑sheet row. Two LLM judges decide 9 clauses on the reply and note, blind to model and version; both must agree. Either judge can run on a model you choose.

Judge models

Opus 5.5 through Claude Code and GPT‑5.6 Sol through Codex, at high effort, with no tools. Before grading, both passed a gate: the reference answer and ten near‑misses, each judged right three times.

Judge agreement

The judges split on 49 of 648 clause verdicts (7.6%); the environment's author ruled each, 25 to pass and 24 to fail. After an audit, 7 agreed verdicts were corrected to match rulings on the same wording.

Formula

Reward = the sum over layers of the layer's share × the fraction of its checks passed, from 0 to 1.

One payment drove the spread; no attempt fully solved it with only the request

With only the request, the 24 attempts scored 37–99%, median 78%. Missing the payment worth 20% cost an attempt 20 points (Opus 5.5's three scored 99, 96 and 79%), and one model's three attempts lay 23 points apart. Told the four style rules, 2 of 24 fully solved it.

Each dot is one attempt, three per model; the line marks the median.

Three attempts per model and version; the table gives each model's mean.

With a hint: the request plus a paragraph saying the company's expectations show up as patterns across its documents, with three worked examples from an unrelated company.

Told the rules: here, only the four house‑style rules the past replies follow, with no figures, dates or identifiers; nothing about the payments.

What the models got wrong

Counted over the 24 attempts with only the request, 8 models × 3, unless a line says otherwise.

1. A search taken as the record. 10 of 24 attempts misread the records on one payment and wrote it up untested, losing 20 points; Opus 5 and GPT‑6 Astra never did. 19 of 25 misses by attempts that saved files came after reading a past case pointing the right way.

2. A second payment, the total wrong. All 3 of GPT‑6 Astra's attempts and 1 of GPT‑6.1 Sol's misread a second payment and paid the wrong total; no Anthropic attempt did.

3. A third payment's record, never found. 8 of 24 attempts missed the record that settles a third payment, some taking an earlier write‑up's word instead: all 3 of GPT‑6 Luna's and 5 of the two Sonnet models' 6. The other five models found it in all 15.

4. House style. 22 of 24 attempts wrote an item, order, return or charge differently from past replies and notes. Told the four style rules, only 3 of 24 slipped on them.

5. The rules don't cover the costliest miss. Told the four style rules, 8 of 24 attempts still misread the payment worth 20%.

What an attempt costs

Length

With only the request, the median attempt took 5 minutes and about 93 tool calls (36 to 201 across the 24), and about 2.1 million tokens, cached input included.

Cost

Recorded for 15 of the 24 attempts with only the request, all Anthropic's, as Claude Code's own figure at list prices; the GPT models' subscription records none, so we give no median. That's the agent model only; the judges' calls aren't included.

What you license

With a licence

The hub with its 66 files and desk tools, the request in all three versions, the grader with its judge clauses and their gate, the answer key and its reference answer, and the grader's report on each of the 72 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran on one Mac under each harness's own sandbox, outside Harbor and any container.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• Three attempts per model and version. With only the request, a model's attempts lay up to 23 points apart, mostly on one payment, so first attempts alone would have named a different leader.
• 9 of the 61 checks depend on LLM judges; their 49 splits across the 72 attempts were settled by the author's written ruling, and 7 agreed verdicts corrected.
• The 20% weight on one payment was set with the results in hand, as the largest at which the models' order stays reliable across their attempts (reliability 0.78 at 20%, 0.72 at 25%).
• The three keep‑out checks have no gate, so a reply that says almost nothing still passes them: 14.4% of the reward. The audit found no attempt that exploited it.
• An audit on 13 October fixed five grading faults (0.3 to 3.2 points an attempt) and made an empty attempt score 0 instead of about 20%; all attempts were regraded.
• GPT‑6 Luna's nine attempts were added a day later, on the same frozen files and judge wording. Two of GPT‑6 Astra's first attempts hit a subscription's usage limit and were repeated.
• Doing nothing scores 0 and the reference answer 1 (the grader's self‑test). No red‑team attempts yet.