AsapiThe AI Agent Trials
Asapi / Catalogue / Coffee web‑shop order desk through a four‑act Christmas peak
Environment datasheet · Small business

Coffee web‑shop order desk through a four‑act Christmas peak

ID tb‑tasks/thornbury‑orders · version 1 · results as of 18 November 2026

As a coffee roaster's order desk through one Christmas season, in four acts, the agent runs the daily paperwork through jobs it schedules, answers every customer case by the policy, claims late parcels, and manages stock, payments and a faulty roast.

Checks
207–213, over four acts, with gates
Tested
5 models, one attempt each
Best
94% (Opus 5.5)
Median
90%
Full solves
0 of 5
Median attempt
306 min, 1903 tool calls
What it tests

A Christmas season at a coffee roaster's order desk, run partly by the agent's own jobs

Long‑horizon work

Up to 48 sessions per act keep only the agent's files, and the daily paperwork runs through jobs it schedules between sessions.

Stateful tool use

Scheduled jobs keep running between sessions, so a mistake in one goes unseen unless the agent checks its own records.

Planning ahead

Second shifts must be booked two working days ahead, and coffees running short taken off sale before stock runs out.

When to act or escalate

Restraint checks fail action the records don't support, such as moving a region on a carrier's bulletin alone.

Catching errors

Customers pick the wrong case category about one time in five, and carrier bulletins are late, partial and once false.

How it works

A simulated web shop, a warehouse, two carriers and 800 subscribers

The agent's world

A shell in a container, one fresh session per step, with one orders tool to a simulator that refuses without saying why.

The simulator

A web shop with about a hundred orders a day, doubling at peak, plus subscriptions, payments, a warehouse, a roastery, two carriers and a helpdesk.

Sessions

A setup, then a 15‑minute session each working day and short ones when a reply lands; scheduled jobs run three times a day.

The reward

The mean over four acts of checks passed, gates counting double; an act that finance takes back partway scores 0.

Results

Four of the five scored 86–94%, and the fifth 50%

Each dot is one attempt, and the line marks the median.

MedianFully solved
5 modelsOne attempt each90%0 of 5
ModelReward
Opus 5.5Anthropic94%
Sonnet 5.5Anthropic92%
GPT‑6.1 SolOpenAI90%
GPT‑6 AstraOpenAI86%
GPT‑6 LunaOpenAI50%

No attempt passed all four acts; the gate on cases and despatch failed in 14 of the 20 acts.

What models get wrong

They give one reply to two kinds of case, overspend, and let stock run out

Catching errors

  1. Every attempt answered customers asking after a slow or held‑back parcel with one fixed reply, giving 15 to about 40 of them a wrong remedy in an act.

  2. When parcels hand‑labelled during a printer outage came back in act 2, four attempts left them unreplaced or asked for an address check; the fifth had been stopped.

When to act or escalate

  1. Six of the 20 acts went over the service‑cost allowance, mainly by remaking or shipping trade orders early and sending replacements twice.

Planning ahead

  1. In 7 of the 15 acts that tested it, a coffee about to run short came off sale only after it ran out, or never.

Stateful tool use

  1. One left a mistake in a scheduled job that no session checked against the records, and finance took the desk back partway through act 2.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $3.01 to $174.57 for the agent model; grading is a script, with no model calls.

What's in it

The simulator and its records, the orders tool and scheduler, the replaying grader, a scripted reference solution, and reports on all 20 acts.

Format

A Harbor task per act in a Docker container, with the simulator running as root beside the agent.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.