Coffee web‑shop order desk through a four‑act Christmas peak
ID tb‑tasks/thornbury‑orders · version 1 · results as of 18 November 2026
As a coffee roaster's order desk through one Christmas season, in four acts, the agent runs the daily paperwork through jobs it schedules, answers every customer case by the policy, claims late parcels, and manages stock, payments and a faulty roast.
- Checks
- 207–213, over four acts, with gates
- Tested
- 5 models, one attempt each
- Best
- 94% (Opus 5.5)
- Median
- 90%
- Full solves
- 0 of 5
- Median attempt
- 306 min, 1903 tool calls
A Christmas season at a coffee roaster's order desk, run partly by the agent's own jobs
Long‑horizon work
Up to 48 sessions per act keep only the agent's files, and the daily paperwork runs through jobs it schedules between sessions.
Stateful tool use
Scheduled jobs keep running between sessions, so a mistake in one goes unseen unless the agent checks its own records.
Planning ahead
Second shifts must be booked two working days ahead, and coffees running short taken off sale before stock runs out.
When to act or escalate
Restraint checks fail action the records don't support, such as moving a region on a carrier's bulletin alone.
Catching errors
Customers pick the wrong case category about one time in five, and carrier bulletins are late, partial and once false.
A simulated web shop, a warehouse, two carriers and 800 subscribers
The agent's world
A shell in a container, one fresh session per step, with one orders tool to a simulator that refuses without saying why.
The simulator
A web shop with about a hundred orders a day, doubling at peak, plus subscriptions, payments, a warehouse, a roastery, two carriers and a helpdesk.
Sessions
A setup, then a 15‑minute session each working day and short ones when a reply lands; scheduled jobs run three times a day.
The reward
The mean over four acts of checks passed, gates counting double; an act that finance takes back partway scores 0.
Four of the five scored 86–94%, and the fifth 50%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 94% |
| Sonnet 5.5Anthropic | 92% |
| GPT‑6.1 SolOpenAI | 90% |
| GPT‑6 AstraOpenAI | 86% |
| GPT‑6 LunaOpenAI | 50% |
No attempt passed all four acts; the gate on cases and despatch failed in 14 of the 20 acts.
They give one reply to two kinds of case, overspend, and let stock run out
Catching errors
Every attempt answered customers asking after a slow or held‑back parcel with one fixed reply, giving 15 to about 40 of them a wrong remedy in an act.
When parcels hand‑labelled during a printer outage came back in act 2, four attempts left them unreplaced or asked for an address check; the fifth had been stopped.
When to act or escalate
Six of the 20 acts went over the service‑cost allowance, mainly by remaking or shipping trade orders early and sending replacements twice.
Planning ahead
In 7 of the 15 acts that tested it, a coffee about to run short came off sale only after it ran out, or never.
Stateful tool use
One left a mistake in a scheduled job that no session checked against the records, and finance took the desk back partway through act 2.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $3.01 to $174.57 for the agent model; grading is a script, with no model calls.
What's in it
The simulator and its records, the orders tool and scheduler, the replaying grader, a scripted reference solution, and reports on all 20 acts.
Format
A Harbor task per act in a Docker container, with the simulator running as root beside the agent.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials