The materials planner

Keep a coffee roaster supplied through two sixteen-week seasons with no memory between days: buy coffee and packaging, plan the roasting, price the products, and never lose a customer.

Setting: Coffee roaster, materials planning
Deliverable: Purchase orders, weekly roasting plans, monthly pricing advice and supplier mail, through a planning tool

Long-horizon judgementUnstated requirementsConflicting sourcesMoney judgement

Models

ran the standard setup

91%

Best score

Opus 5.5

70%

Median score

across those models

82 min

Typical run

median time, standard setup

The task

A made-up coffee roaster supplies a supermarket, a cafe chain and a private-label brand every week. The agent is its materials planner for sixteen simulated weeks of summer and again of autumn, woken each working day that mail arrives, with only a workspace to remember by. It buys green coffee and packaging, files the weekly roasting plan and a monthly pricing advice, and answers the mail: call-offs, suppliers' letters, a sales manager's tips, engineering's notices.

The prompt only asks that materials are there when the plan needs them, that nothing is bought the store cannot hold, and that the prices are right. The real test is reading the seasons and the disruptions from the plant's own history and a sister plant's records, sizing each one, and acting early enough without filling a small store.

What it tests

Judgement over weeks

Orders take weeks to arrive, several matters are open at once, and every lesson has to be written down to survive the next session.

What the job really needs

Nothing says that demand and supply are seasonal or how big a disruption will be; the records show it.

Knowing which source wins

A sales forecast, a manager's tips, suppliers' letters and two plants' records each tell part of the story, and some of it is wrong.

Money judgement

The pricing advice must be costed from the coffee actually in store, and a supplier's holding fee is cheap next to a lost order.

Challenges

1Seasons nobody mentions

Like a real plant, the peaks and dips are only in past despatches and receipts.

2Disruptions without their size

A packaging maker's shutdown, a roaster overhaul, a port closure and a supermarket promotion arrive in the mail; only a larger sister plant's history shows what such events did, at a different scale.

3Tips and forecasts

A sales manager passes on prospects, only some of which become orders, and the sales forecast runs high.

4A small store

Coffee the store cannot take goes outside and spoils within days, so buying too early costs as much as buying too late.

5Customers who leave

A customer ends its agreement after repeated short deliveries, and the season ends there.

6No memory between days

Each session starts fresh, so the workspace is all that carries over.

Results

Model	Standard	Time (standard)	Cost (standard)
Opus 5.5	91%	116 min	$41.09
Sonnet 5.5	50%	48 min	$10.42

One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

No model passed both seasons. Opus 5.5 scored 90.9% over two runs: it ran all four of its seasons to the end and passed every measure in both summers, building its own tools from the records and handling the shutdown, the overhaul and the port closure in its buying.
Opus 5.5 failed both autumns on the monthly pricing advice: it counted coffee the port closure was holding back as arriving in time, although its own notes listed those deliveries as late.
Opus 5.5 and Sonnet 5.5 both left the prospect that became a real order uncovered in seven of their eight seasons, buying nothing and asking no supplier to hold coffee; by the time the order came, the coffee that could have arrived in time was sold.
Sonnet 5.5 scored 49.8%: all four of its seasons ended early, in weeks 8 to 14, when a customer left after one material ran out that it had stopped tracking, such as packaging used up faster by roasting ahead, or coffee held up by the port closure.
Both models waited for engineering's confirmation before ordering a new product's packaging in three of four autumns, and missed its first weeks of orders.
Both models resisted the high sales forecast, the prospect that never came and a shipping notice later withdrawn.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Simulated plant: the materials planning desk of Thornbury Roasting & Packing at Mill Lane, a made-up coffee roaster with three customers on supply agreements, three green-coffee importers and two packaging makers, run by a hidden simulator for two sixteen-week windows (summer, from 2026-06-01; autumn, from 2026-09-07), each a scenario of its own
Agent's interface: Two ways, one per agent family. Codex CLI 0.160.0 (the OpenAI models) through harbor: a terminal in a fresh Docker container (Ubuntu 24.04, Python 3, the SQLite command line; 2 CPUs, 4 GB), one fresh session per step as an unprivileged user. Claude Code (Opus 5.5, Sonnet 5.5) through the task's local driver: a fresh claude -p process per session on a workstation, outside Docker, working in the trial's own folder under /tmp, with the trial's copies of /data, /docs and /workspace (the prompt and the docs name those folders' paths) and desk on its path. Both drive the same simulator
Documents: At the setup session, in /docs, 3 Markdown files (the company, the week and what is measured; the desk command; a copy of the instruction). In /data, summer 239 files: 98 CSV, 79 mail (.eml), 53 text, 8 Markdown and 1 SQLite: the stores and purchasing policies and 3 supply agreements (Markdown), product tables (6 CSV), the supplier register, freight and duty, 75 monthly price lists (CSV and text) and 61 past supplier letters, purchasing records (5 CSV) and 23 past advisories, the commercial manager's 15 past letters, 4 sales forecasts and customers' purchase orders, a sister plant's database (SQLite, with its schema), Mill Lane's scanned green-store ledger (27 monthly text pages), pack ledger, roast log, production, despatch, yard book, weather and loss account (CSV), store map and first Monday report (Markdown), and the first day's 3 mails. Autumn 271 files: 108 CSV, 94 mail, 59 text, 9 Markdown, 1 SQLite (a fourth agreement, 84 price lists, 73 supplier letters, 26 advisories, 20 commercial letters, 5 forecasts, 30 ledger pages, 1 first-day mail). Each day the simulator adds mail, the Monday reports and the records' new rows
Applications: One command, desk: status, inbox, order FILE (purchase-order lines), plan FILE (the roast-and-pack plan for up to eight weeks ahead), advise FILE (the month's pricing advisory, while its window is open), mail VENDOR SUBJECT FILE (write to a supplier's sales desk), stock with green, pack or fg (a storeman's count, at most two a session), show with orders, holds, plans, advisories or mail and done (end the session). Python 3 and the SQLite command line read the records. No other apps
Web: Not needed: nothing the task asks about is online. In harbor the container can reach the internet and Codex's web search was turned off; the local Claude Code runs had the host's internet and Claude Code's own web tools on, and no board session used them
People: None who answer free questions. The simulator's code, with no language model, plays the suppliers' sales desks (they answer a letter in one to three working days, line by line, with availability, price, earliest delivery and holds), the customers (call-offs, purchase orders, calendars), the commercial manager (prospects and tips), the storeman, engineering and finance, whose mail arrives on fixed rules; the same commands give the same replies in every run
Time: An in-world calendar: deliveries booked in on Mondays, the plan runs on Tuesdays, call-offs on Thursdays, despatch on Fridays. The clock is frozen during a session; desk done (or the session's end) runs time day by day to the next working day that brings mail, which opens the next session. One untimed setup session, then sessions of 15 minutes of desk time; up to 80 sessions in sixteen weeks; the window ends early if a customer terminates its agreement, if that has become unavoidable, or if finance stops buying
Actions: Place purchase orders (the system accepts or refuses each line at once, without saying why), file plans and advisories, write to the suppliers' desks (including asking for holds), and count a store. Nothing else in the records can be changed
Deliverable: Purchase-order lines (CSV), the weekly roast-and-pack plan (CSV: week, SKU, batches), the monthly pricing advisory (CSV: SKU, break-even, review flag) and letters to suppliers, all filed with desk; the agent's own notes in /workspace carry over between sessions

← PreviousThe parts buyer All 34 tasks