Coffee web‑shop order desk through a four‑act Christmas peak

Hundreds of small duties a day, kept by notes and scheduled jobs alone

Role

The order desk for a coffee roaster's web shop, subscriptions and trade counter, over four separate acts of two to three weeks each.

Request

Keep the promises: every customer's coffee delivered as fresh as the shop promises and subscribers' bags on their day; answer every case as the service policy says; keep service costs within the allowance.

Deliverable

The desk's actions, filed in its sessions and by the scheduled jobs it leaves in its workspace: work orders, manifests, voids, case remedies, claims, notices, holds, payments, stock flags, quotes and mail.

What the reward depends on

Paperwork run every working day; cases closed with the policy's remedy in time; regions moved only on the despatch policy's test on delivery‑scan transit times; nothing needless done; costs within allowance; the faulty batch held in time.

Where the requirements are

In two years of records growing daily: policies, the warehouse procedure, carriers' terms, scan files and bulletins, orders, cases and mail, 3,892 to 4,430 files per act at setup.

An order‑desk simulator per act, woken by mail, with scheduled jobs

Interface

A shell in a Docker container as an unprivileged user, one fresh session per step, with one orders tool over a socket to a root‑owned simulator. The ordering, despatch and payment systems refuse without saying why.

Scheduled jobs

Executables left in the workspace's jobs folder run at three points of every working day with only the paperwork commands; the clock stands still while the agent works.

The simulator

The web shop with about a hundred orders an ordinary day, doubling in the peak, subscriptions, trade schedules, a payment gateway, a warehouse with pick capacity, a roastery, two carriers and a helpdesk.

Simulated people

Code plays the customers, about 800 of them active subscribers, who pick the wrong case category one time in five; the carriers' desks, whose bulletins are late, partial and once false; and the warehouse, roastery and trade accounts.

Sessions

Each act has up to 30 to 48 sessions: a setup untimed on the desk's clock, a session of 15 desk‑clock minutes each working day and short ones when a reply lands. Only the workspace persists.

Limits

90 minutes of agent time for the setup session and 20 for each later one.

Checks and gates per act, graded by replaying every action

Structure

75 listed checks per act, usually 46 to 55 run, in families for the rules, restraint, evidence, numbers, deliverables and cross‑checks. Each gate is worth 2 points, every other check 1.

Gates

An act passes only on its promise and subscription‑day rates, cases and despatch at the levels the policies set, service cost within the allowance and no early stop by finance over runaway costs; the task passes only if all four acts do.

Who decides

After each act, the grader replays the logged commands in a fresh simulator, checks the replay reproduces the attempt, and grades it; no LLM judge. Audit fixes of 28 and 29 October were applied by regrading every act.

Formula

Reward = the mean over the four acts of points passed / points run. Restraint checks fail actions the records don't call for.

Every model scored 85% or more but one, and none passed all four acts

Opus 5.5 scored 94%, Sonnet 5.5 92%, GPT‑6.1 Sol 90% and GPT‑6 Astra 86%, and GPT‑6 Luna 61%. No attempt passed all four acts: the cases‑and‑despatch gate failed in 14 of the 20 acts.

Each dot is one attempt; the line marks the median.

One attempt per model, five models, each act packaged separately and averaged. Two models' earlier attempts through a local driver were replaced by attempts under the same rules as the others.

A fifth package covering the whole season isn't on the board.

What the models got wrong

Counted over the 5 models, one attempt each, across 20 acts.

1. One reply for two cases. Every attempt answered customers asking after a slow or held‑back parcel with one fixed reply rule, giving 15 to about 40 of them the wrong remedy in an act; it was the largest group of failed cases in 11 acts.

2. Over the allowance. 6 of 20 acts, in 3 of 5 attempts, went over the service‑cost allowance, mainly by remaking or shipping trade consignments early and sending replacements twice.

3. Stock that ran out. In 7 of the 15 acts that tested it, a coffee about to run short came off the shop only after it had run out, or never.

4. Hand‑labelled parcels. In act 2 no attempt replaced the parcels hand‑labelled during a printer outage when they came back: 4 left them unreplaced or answered their customers with an address check, and the fifth's desk had already been stopped by finance.

5. A silent filter mistake. 1 of 5 left a mistake in its scheduled job that no session checked against the records, and finance took the desk back on 6 November, partway through act 2.

What an attempt costs

Length

The median attempt took 306 minutes of agent time over its four acts and 1,903 tool calls (1,487 to 2,268 across the 5).

Cost

The median attempt cost $56.88, from $3.01 to $174.57 across the 5, $351 in all: Harbor's figures from the logged tokens at list prices for the Codex attempts, Claude Code's own for the Claude Code attempts.

What you license

With a licence

Version 1, the four acts: the simulator and its records, the orders tool and scheduler, the grader with replay, a scripted reference solution, and the grader's report on each of the 20 acts, as baseline results.

Format

A Harbor task per act in a Docker container, with the simulator running as root beside the agent.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The simulator, records and grader aren't published.

What we haven't measured, and what's uneven

• Doing nothing after the setup session scores 48% to 53% per act, mostly from restraint checks that pass when nothing is done, though finance stops every act and none passes.
• The scripted reference solution passes all four acts but scores 97% in act 3, not 100%.
• One attempt per model, so we haven't measured how much attempts vary here.