Two‑site spares buying through supplier part withdrawals

Six withdrawn parts, two sites, one monthly budget

Role

Group buyer for the spare‑parts stores at a coffee roaster's two sites, for 12 simulated weeks from January to March. It's the same plant as in Two‑site spares pick lists and stock plans.

Request

Make sure the parts each repair takes are on the shelf when needed, at the site that needs them, within the purchasing policy, as suppliers withdraw parts; don't buy what the stores already hold.

Deliverable

No file. Bills of materials, supplier mail and purchase orders made through one command‑line tool, judged by repairs that waited for a part and by wasted purchase lines.

What the reward depends on

For each withdrawal, the one brochure row that meets every limit, accepted and ordered for every affected machine at both sites, early enough for the supplier's real lead time; meanwhile, routine stock topped up, including every part of two‑part repairs.

Where the requirements are

Limits sit in the engineering standard, installation data, room register, manuals and one old work‑log remark. The two sites name parts differently. Real lead times, and one supplier that serves one site only, show in the purchasing records.

A purchasing simulator with change control and 12 scripted people

Interface

A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, SQLite; 2 CPUs, 4 GB); one command‑line tool talks to a simulator running as root. Our attempts ran as multi‑step Harbor trials on EC2, one step per session, driven by Claude Code or Codex CLI.

Commands

Status, inbox, file a bill of materials with a site's lead technician, mail a supplier's desk, place orders, show records, count a drawer (two a session), and end the session. Orders are refused without a reason; uncoded parts can't be ordered.

Files

219 at the start: policies, the group's engineering standard, both sites' records (one as files, one a database), a year of orders, deliveries and 68 invoices in suppliers' own formats, the supplier register and catalogue, 100 past supplier mails and past bills of materials.

Scripted people

12 roles, no LLM: a lead technician and a storeman at each site, the stores controller, and 7 suppliers' sales desks. A desk reads only the part numbers in a mail, replies in one to three days and never judges suitability.

Withdrawals

Six parts are withdrawn on fixed days, each announced by a letter, a mailing, a refused order, a storeman's request or the deprecations list, usually with a brochure. In five of six brochures the supplier's own suggestion breaks a site limit.

Change control

The lead technician answers a bill of materials in two working days; one objected line rejects it. A comment names the failed property, giving a value only for the one limit no document states. Accepted parts get a stock code the next working day.

Sessions

One opens at the end of each working day with mail, at most one a day: 38 to 47 in each of our 5 attempts, of 61 declared. Setup is untimed in the world; later sessions allow 10 minutes of writes, or 15 with a brochure or the Monday report.

Memory

Only the workspace folder carries over: each session starts the harness with a new memory folder. Between sessions a root verifier stops leftover processes and deletes Codex's and other agents' memory folders.

Network

The container could reach the internet, though nothing the task needs is online. Claude Code's web reader was on and unused; Codex's web search was off, and no command reached the web.

The plant

Machines fail and draw parts; storemen mail only when a bin empties. True lead times can differ from quotes; a van brings listed parts from the other site in hours; a monthly budget and drawer sizes bind.

What the simulator decides

Failures, letters and notices land on their days whatever the agent does; the agent changes what technicians find and what people reply. The same commands give the same 12 weeks in every attempt.

61 repairs that mustn't wait, and one unscored waste check

Structure

62 checks: 61 counted repairs, 17 at machines hit by a withdrawal and 44 routine, each passing if it didn't wait more than a shift for a part; and one waste check that carries no points. A part the van brings isn't a wait.

Left out

Six repairs that fell too soon after a withdrawal for any replacement to be agreed and delivered aren't counted, the same six in every attempt.

Waste check

Passes with at most one purchase line that bought a replacement, or a withdrawn part once its withdrawal had reached the buyer, for a site whose shelf and open orders already covered the rest of the 12 weeks.

Formula

Reward = repairs that didn't wait / 61, from 0 to 1. One repair is worth about 1.6 points.

Grader's pass

The grader also reports a pass: no repair waited and the waste check holds. 2 of 5 attempts (5 models, one each) passed, both full solves.

Who decides

Code alone: no LLM judge, and no result settled by ruling. After the last session the grader replays the logged commands through a fresh simulator, checks that the replay reproduces the attempt's trace hash, and grades the replayed world.

Regrade

After an audit on 13 October, the owner ruled that a line bought before its part's withdrawal reached the buyer isn't waste. All 5 attempts were regraded from their own logs: Opus 5.5's waste check now passes; no reward or pass changed.

Doing nothing

37 of 61 repairs wait, and the attempt stops after week 6 with 39%, which fails. The scripted reference solution scores 1.0 in 43 sessions.

Two full solves in five; every lost check was late stock

With only the request, 2 of 5 attempts (5 models, one each) were full solves: GPT‑6 Astra's and GPT‑6.1 Sol's. All 5 chose the right replacement for all six parts at the first filing; the 9 repairs that waited, across 3 attempts, needed right parts not yet on the shelf.

Each dot is one model's attempt; the line marks the median.

One attempt per model, with only the request. There's no hint or told‑the‑rules version: every attempt had the same request in every session.

Because an idle agent scores 39%, rewards sit close together: the 5 attempts scored 90% to 100%.

What the models got wrong

Counted over the 5 models, one attempt each, with only the request: 9 repairs waited in all.

1. One site of two. 2 of 5 attempts (Opus 5.5, GPT‑6 Luna) first filed replacement kits only for the site whose records name parts in words, not the one using numbers: 3 repairs waited. Opus 5.5 found its gap two weeks later, then held both orders for next month's budget.

2. Restocking on empty bins. A drive‑belt repair takes a belt and two pins. 2 of 5 attempts (Sonnet 5.5, GPT‑6 Luna) reordered belts when a bin emptied and missed that the pins were down to one, which no mail flags: 4 repairs waited. The other 3 reordered pins in time.

3. Orders that wouldn't arrive. GPT‑6 Luna's other 2 waits: it relied on an order its supplier had written to cancel, and left a site with fewer belts than one repair takes. It also reordered from a supplier shipping by sea while faster ones were listed.

What an attempt costs

Length

With only the request, the median attempt took 61 minutes of agent time over its sessions and 410 tool calls (322 to 501 across the 5), and about 15 million tokens, cached input included.

Cost

Recorded at list prices for only 2 of the 5 attempts, so we give no median: Claude Code's own figures, on Bedrock, $10.45 for Opus 5.5 and $6.16 for Sonnet 5.5. Agent model only; nothing else in the environment calls a model.

OpenAI's estimates

The three OpenAI attempts ran on a plan sign‑in, not billed per call. Harbor's estimates from their tokens, at prices it assumed, run from $0.45 to $28.04; they aren't comparable with Claude Code's figures.

What you license, and what's public

Format

Packaged as a Harbor task on request; it isn't one of our three downloadable samples. Our attempts ran on the task's own multi‑step Harbor build.

With a licence

The simulator with both sites' records, the suppliers, withdrawals and scripted people, the request, the grader with its replay check, the scripted reference solution, and the grader's report on each of the 5 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, simulator and grader aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model, each in its maker's harness (Claude Code or Codex CLI), so we haven't measured how much attempts vary, and model and harness effects are mixed.
• An idle agent earns 39%, though the grader fails it: 37 of 61 repairs wait before the stop after week 6. So rewards sit close together near the top.
• Wasted buying carries no points; it only decides the grader's pass, which allows one wasted line.
• Every attempt faced the same plant: the same failures in the same weeks, and the same withdrawals.
• Costs aren't measured the same way: Claude Code reports its own; the OpenAI figures are Harbor's estimates from tokens at its assumed prices, on a plan sign‑in, not bills.
• No red‑team attempts yet.