Coffee roaster materials planning across two seasons

Buy ahead of seasons and disruptions nobody sizes for you

Role

Materials planner at a coffee roaster's plant that supplies a supermarket, a cafe chain and a private‑label brand under supply agreements, for two separate 16‑week windows, from June and from September.

Request

Have the green coffee and packaging in store when the plan needs them, make what the customers' weekly call‑offs ask for, buy nothing the store can't hold or the plant can't use in time, and tell the business each month what each product costs to make.

Deliverable

No file. Purchase orders and supplier holds, a roast‑and‑pack plan by the Friday before each week, a pricing advisory each month (each product's break‑even and review flag), and mail to suppliers, all through one command‑line tool.

What the reward depends on

Deliveries in full and on time with no customer lost; each disruption met early; the sales tip that becomes an order covered and false signals ignored; nothing spoiled or lost; break‑evens costed from the coffee actually in store or truly due.

Where the requirements are

No document says demand is seasonal or how big a disruption will be; two years of despatches and a larger sister plant's records show it, at a different scale. The policies state the costing method and allow a supplier hold.

A plant simulator per window, with scripted customers, suppliers and events

Interface

Two ways. Codex CLI ran in Harbor, in Docker; Claude Code ran through the task's local driver on a workstation, outside Docker: a fresh process per session in the attempt's own folder, with copies of the documents, records and workspace, and the planning tool on its path.

Harbor package

Each window is a multi‑step Harbor task in Docker: one fresh session per step as an unprivileged user, with the simulator running as root. The three Codex CLI attempts, and the reference‑solution and empty‑attempt gates, ran there.

Commands

Status, inbox, place orders, file the plan for up to eight weeks ahead, file the month's advisory while its window is open, mail a supplier's desk, count a store (two a session), show records, and end the session. Orders are refused without a reason.

Files

Three guides, plus 239 files in the summer window and 271 in autumn: policies, supply agreements, product tables, two years of importers' price lists and letters, purchasing records, past advisories, the commercial manager's past mail, forecasts, the plant's records and the sister plant's database.

Scripted people

No LLM. Code plays five supplier desks (three coffee importers, two packaging makers), three customers, the commercial manager, engineering, a shipping line, the storeman and finance. Desks answer in one to three working days and hold coffee if asked, for a fee.

Customers

Weekly call‑offs and purchase orders. A customer ends its agreement after repeated short or late deliveries, and the window ends there. It also ends once that is unavoidable, or if finance stops buying when goods lost run far past the allowance.

Plant week

Deliveries are booked in on Mondays, with overflow to a yard where green coffee spoils in days; the filed plan roasts the oldest coffee first on Tuesdays; customers call off on Thursdays; despatch runs on Fridays.

Sessions

One opens each working day that brings mail: up to 80 per window, 10 to 52 in our 14 windows. Setup is untimed; later sessions allow 15 minutes on the desk's clock. Agent time: up to 90 minutes for setup and 20 a session in Harbor, 25 a process locally.

Memory

Only the workspace is meant to carry over. Each session was a fresh process with no conversation resumed. Harbor also stops leftover processes and clears the agent's memory folders after each; the local driver skips both, which need a separate user.

Network

The container and the workstation could reach the internet, though nothing the task needs is online. Codex's web search was off; Claude Code's web tools were on, and no session used the web.

Events and signals

Summer brings a roaster overhaul and a packaging maker's shutdown; autumn a crop shortage, a port closure, a promotion, a new product and the Christmas peak. Each window has a sales tip that becomes an order, one that doesn't, a forecast that runs high and a closure warning later withdrawn.

What the simulator decides

Demand by each customer's season and events, supply by each origin's seasonal lead time, and every reply. The history files come from the same engine, run by a scripted earlier planner. The same commands give the same 16 weeks in every attempt.

Up to 168 weighted checks over two windows, with three gates in each

Structure

Summer has 81 checks worth 50 points and autumn 87 worth 53.75: plans and advisories filed, each product's break‑even each month, costing method and review flags, each disruption and tip handled, restraint, nothing lost, each customer's deliveries, and three gates.

Weights

From 0.25 points, for one product's break‑even in one month, to 2, for each of the three gates and for running the window to its end.

Formula

Window score = points passed / points listed. Reward = the mean of the two windows, from 0 to 1.

Early end

Checks a window never reached fail, except that a month whose advisory wasn't yet due lists only its method check. So an attempt has 114 to 168 checks.

Gates

The grader's pass, per window: deliveries in full and on time at or above a set rate, with no early end; goods lost within an allowance; the three in‑window advisories filed, at most one product‑month out of band, every review flag right. An attempt passes if both windows do.

Who decides

Code alone: no LLM judge, and no result settled by ruling. After each window's last session the grader replays the logged commands through a fresh simulator, checks the trace hash, and grades it, taking true costs from the agent's own lots.

Two readings

A break‑even passes inside a band around the policy's figure, widened to accept the order‑book and first‑in‑first‑out readings of the method; a product near its review threshold passes with either flag.

Doing nothing

A customer leaves in week 5 of either window, for about 2%, which fails. The scripted reference solution scores 1.0 in both windows, in 40 and 38 sessions.

No full solve; two attempts passed both windows

Of 7 attempts (5 models, with only the request), none fully solved it and two passed both windows: GPT‑6.1 Sol's, at 97%, and GPT‑6 Astra's, at 94%. Opus 5.5's two passed both summers and failed both autumns' pricing gate. All six of Sonnet 5.5's and GPT‑6 Luna's windows ended early.

Each dot is one attempt; the line marks the median.

Two attempts each for Opus 5.5 and Sonnet 5.5 and one for each OpenAI model, each a summer and an autumn window, with only the request; the table shows each model's mean. There's no hint or told‑the‑rules version: every attempt had the same request in every session.

Anthropic's attempts ran on 14 and 15 October, just before the task was frozen, on identical files; OpenAI's on 23 and 28 October. A GPT‑6 Astra attempt whose sign‑in failed, and two windows that loaded the wrong window's files, were made again.

What the models got wrong

Counted over the 7 attempts (5 models), 14 windows in all, with only the request. 13 of the 14 rightly ignored the sales tip that came to nothing.

1. The real tip left uncovered. 9 of the 13 windows that reached it neither bought nor held coffee for the tip that became an order; by the order, the coffee that could arrive in time was sold. Opus 5.5 read the rules as banning the hold they allow.

2. A closure missed in the pricing. Both of Opus 5.5's autumn windows priced 2 or 3 of 13 products below their true break‑even in November, counting coffee the port closure was holding back as due, though its buying treated those lots as late. Each failed the pricing gate.

3. A material left to run out. All 4 of Sonnet 5.5's windows ended early, in weeks 8 to 14, when something it had stopped tracking ran out: cartons once, after its note said to top them up, and coffee three times.

4. Counting stock already called off. Both of GPT‑6 Luna's windows ended early, in weeks 7 and 4, after plans sized from a count of finished goods that still held the cases that week's despatch would take, so lines shipped short.

5. Waiting for a confirmation. In 3 of the 6 autumns that reached it, the agent waited for engineering to confirm a new product's line before ordering its packaging, though the listing gave the start week, and missed its first weeks; the other 3 supplied it from the first.

6. Promised dates trusted. GPT‑6.1 Sol and GPT‑6 Astra each planned an autumn week on coffee the port closure held arriving before it did, with a warehouse lot outside the port on offer, and one or two lines went short; both still passed every gate.

What an attempt costs

Length

With only the request, the median attempt took 113 minutes of agent time over both windows (37 to 369 across the 7) and 878 tool calls (323 to 1,117).

Cost

Every attempt ran on a plan sign‑in, not billed per call. Claude Code's own figures, at list prices, for its 4 attempts: a median of $25.47, from $8.12 to $43.96, and $103 in all. Agent model only; nothing else in the environment calls a model.

OpenAI's estimates

Harbor's estimates from the tokens of the 3 Codex CLI attempts, at prices it assumed, run from $0.37 to $107.32; they aren't comparable with Claude Code's figures.

What you license, and what's public

Format

Packaged as a Harbor task on request, one per window; it isn't one of our three downloadable samples. Codex CLI's attempts ran on the task's own multi‑step Harbor build; Claude Code's ran outside Harbor, through a local driver.

With a licence

Both windows' simulators with the plant's and the sister plant's records and the scripted customers, suppliers and events, the request, the grader with its replay check, the scripted reference solution, and the grader's report on each of the 7 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, simulator and grader aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• Anthropic's 4 attempts ran outside a container: Claude Code on a workstation, beside the simulator, grader and reference solution, with no separate user between them. Nothing stopped a session reading them; an audit of the transcripts found none read the simulator, answer key or control socket.
• In those local attempts, the memory scrub and process kill between sessions didn't run; each session was a fresh process with no conversation resumed, and no session's logged tool calls wrote to Claude Code's memory folders.
• Two attempts each for Opus 5.5 and Sonnet 5.5, one for each OpenAI model, each in its maker's harness, local or in Harbor, so model and setup effects are mixed. Opus 5.5's scored 90% and 92%; Sonnet 5.5's 63% and 36%.
• One restraint check, on not buying after the sales forecast, counts all coffee bought for a set period after it, even for a firm order, later weeks or closure cover. It failed in three of Sonnet 5.5's windows and in GPT‑6.1 Sol's and GPT‑6 Astra's summers; the owner kept it.
• An early end fails every later check, except that a month whose advisory wasn't yet due lists only its method check, so a window that ends early scores over fewer points. The owner kept this. With both issues fixed, Sonnet 5.5's mean would be about 49%, not 50%.
• One cause can fail several checks: an uncovered tip fails up to four in a window. The agent's guide calls the score the share of checks met, though checks weigh 0.25 to 2 points.
• The importers' desks read a letter naming a coffee and a quantity beside the word “hold” as a hold request, even when it said it wasn't one; GPT‑6.1 Sol and GPT‑6 Astra both opened holds this way.
• No red‑team attempts yet.