Coffee roaster materials planning across two seasons
ID tb‑tasks/thornbury‑supply · revision 3 · tested 14 to 28 October 2026 · regraded 18 November 2026
As materials planner at a coffee roaster with three contract customers, the agent buys green coffee and packaging and files a weekly roast‑and‑pack plan and a monthly pricing advisory, through a summer and an autumn window of 16 weeks each. A customer let down too often ends the window.
- Checks
- 112–166, 166 when both seasons run to the end
- Tested
- 5 models, 1 or 2 attempts each
- Best attempt
- 98% (GPT‑6.1 Sol)
- Median
- 90%
- Full solves
- 0 of 7
- Median attempt
- 113 min
Coffee and packaging planned through a summer and an autumn of 16 weeks
Long‑horizon work
Each 16‑week season runs up to 80 fresh sessions keeping only the agent's files, and a customer let down too often ends it early.
Stateful tool use
Stock, orders and holds carry over week to week in each season, and a material the agent stops tracking can run dry.
Planning ahead
Coffee and packaging must be bought ahead of seasonal demand and disruptions, within storage caps and shelf life.
When to act or escalate
Preparing for the sales manager's real tip is rewarded, while buying on a tip that comes to nothing or a withdrawn warning fails.
A plant simulator, three customers, five supplier desks and two seasons
The agent's world
One planning tool to a simulator: orders and holds, a weekly roast‑and‑pack plan, a monthly pricing advisory and supplier mail.
The customers
Three, with weekly call‑offs; one ends its agreement after repeated short or late deliveries, and the season ends there.
Events
Each season brings disruptions, a sales tip that becomes an order, one that doesn't, a forecast that runs high and a withdrawn warning.
The reward
The mean of the two seasons' scores, each the share of weighted checks passed, from deliveries and pricing to restraint.
Four attempts scored 90–98%, and three fell to 65% or less
Each dot is one attempt, and the line marks the median.
| Model | Mean reward |
|---|---|
| GPT‑6.1 SolOpenAI | 98% |
| GPT‑6 AstraOpenAI | 95% |
| Opus 5.5Anthropic | 91% |
| Sonnet 5.5Anthropic | 50% |
| GPT‑6 LunaOpenAI | 27% |
GPT‑6.1 Sol's and GPT‑6 Astra's attempts also passed both seasons' gates.
They leave the real tip uncovered, let materials run out, and misprice a closure
When to act or escalate
69% of the seasons that reached it neither bought nor held coffee for the tip that became an order, and the coffee sold out.
Stateful tool use
All four of Sonnet 5.5's seasons ended early when something it had stopped tracking ran out.
GPT‑6 Luna sized plans from a count of finished goods that still held cases that week's despatch would take, so lines shipped short.
Planning ahead
Both of Opus 5.5's autumns priced products below their true break‑even, counting coffee the port closure held back as due.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.37 to $107.32 for the agent model; nothing else in the environment calls a model.
What's in it
Both seasons' simulators with the plant's and a sister plant's records, the scripted customers and suppliers, the grader, and a reference solution.
Format
Packaged as a Harbor task on request, one per season; Codex's attempts ran in Harbor, Claude Code's through a local driver.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials