The parts buyer

Buy spares for a two-site plant for twelve simulated weeks while six parts are withdrawn by their makers, with no memory between days: pick each replacement, get it approved, and never leave a repair waiting.

Setting: Coffee roaster, maintenance purchasing
Deliverable: Bills of materials, supplier mail and purchase orders, made through a stores tool

Unstated requirementsConflicting sourcesLong-horizon judgementMoney judgement

Models

ran the standard setup

100%

Best score

2 models tied

98%

Median score

across those models

61 min

Typical run

median time, standard setup

The task

A made-up coffee roaster has two sites, one on paper records and one on a maintenance system. The agent is the group buyer for both sites' spare-parts stores for twelve simulated weeks, woken each working day that mail arrives, with only a workspace to remember by. Suppliers write that parts are withdrawn and send brochures of their new ranges; a replacement must be accepted by the site's lead technician before it can be fitted or ordered. Meanwhile machines keep failing, ordinary stock keeps running down, and the monthly budget is fixed.

The prompt only asks that the parts are on the shelf when needed. The real test is checking each replacement against limits spread over several documents, covering every machine at both sites, ordering in time for slow suppliers, and not buying what is already on its way.

What it tests

What the job really needs

The supplier's suggested replacement often fails a limit the site's own records set, and the brochure alone cannot tell the options apart.

Knowing which source wins

Engineering rules, installation records, manuals and the plant's own history each hold part of the answer, and the supplier's advice does not override them.

Judgement over weeks

Several withdrawals are open at once, quoted lead times are not true ones, and every lesson has to be written down to survive the next session.

Money judgement

A fixed monthly budget, express suppliers at several times the price, and a penalty for buying stock the stores already hold.

Challenges

1Recommended but wrong

Like real suppliers, each pushes its own suggestion, which often does not fit the position.

2Two sites, two record styles

The same part is named in words at one site and by number at the other, and both sites' machines need the replacement.

3A supplier that serves one site

One site has to hold stock for the other and send it by van.

4Slow suppliers

Some deliver well after the date they quote, so an order placed when a part is first needed is too late.

5Routine stock never stops

Ordinary parts keep running out while the withdrawals are in hand, and some repairs need two parts at once.

6No memory between days

Each session starts fresh, so notes left in the workspace are all that carry over.

Results

Model	Standard	Time (standard)	Cost (standard)
GPT-6 Astra	100%	61 min	$28.04
GPT-6.1 Sol	100%	91 min	$6.60
Sonnet 5.5	98%	27 min	$6.16
Opus 5.5	97%	39 min	$10.45
GPT-6 Luna	90%	61 min	$0.45

One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

GPT-6 Astra and GPT-6.1 Sol both scored 100% and passed: no repair waited in twelve weeks. Both covered every machine at both sites from their first filing, ordered the slow supplier's parts as soon as they could be ordered, and kept a full set of every part a repair takes at each site.
Every model chose the right replacement for all six withdrawals first time, turning down supplier suggestions that broke a limit; every lost mark was stock that arrived after a repair needed it.
Opus 5.5 (96.7%) and GPT-6 Luna (90.2%) first filed a kit only for the site whose records name parts in words, and the other site's machines waited. Opus 5.5 also held the orders for next month's budget.
Sonnet 5.5 (98.4%) and GPT-6 Luna restocked when a storeman reported an empty bin, and missed that a two-part repair's second part was down to its last; GPT-6 Luna lost three repairs this way, Sonnet 5.5 one. GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 spotted the low part in time.
GPT-6 Luna also relied on an order its supplier had written to cancel, and ordered from a supplier shipping by sea while faster ones were listed.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Simulated plant purchasing: the group buyer for the spares cribs of Thornbury Roasting & Packing, a made-up coffee roaster with two sites, for twelve simulated weeks in which six parts on the pick list are withdrawn by their makers; run by a hidden simulator in the container, the same plant as spares-crib
Agent's interface: A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, the SQLite command line; 2 CPUs, 4 GB), one fresh session each time mail reaches the buyer, as an unprivileged user, driven by the agent's own command-line tool through harbor: Claude Code or Codex CLI
Documents: At the setup session, 219 files: in /docs, 3 Markdown (the company and what is measured, the crib command, a copy of the instruction); in /data, the stores and purchasing policies, the group's engineering standard for replacement parts, Mill Lane's assets and manuals' parts lists, its 2025 and 2026 work logs, Northgate's maintenance export (1 SQLite), the group's part master, cross-references, deprecations, capacities and installed base, a year of purchase orders, deliveries and invoices in each supplier's own format (68 text invoices), the supplier register and catalogue, 100 past mails with the suppliers, past bills of materials and their verdicts, and the first inbox note; 21 CSV, 2 logs, 15 Markdown, 1 SQLite and 180 text files in all. Every working day the simulator adds mail, brochures, notices, verdicts and the sites' records
Applications: One command, crib: status, inbox, bom FILE (file a bill of materials with a site's lead technician), mail VENDOR "SUBJECT" FILE (write to a supplier's sales desk), order FILE (place purchase orders), show orders/boms/mail, count SITE DRAWER (a physical count, at most 2 a session) and done (end the session). The SQLite command line and Python read the records. No other apps
Web: Not needed: nothing the task asks about is online. The container can reach the internet; the plant and its suppliers are made up. Claude Code's web page tool (WebFetch) was on for Opus 5.5 and Sonnet 5.5, and neither run used it; Codex's web search was turned off for GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna
People: None who answer freely. The simulator's code, with no language model, plays the suppliers (letters, brochures and replies to mail, which carry only its subject), the sites' lead technicians (who accept a bill of materials or send it back, sometimes with a comment, in two working days), the stores controller (who re-issues the pick list and raises a stock code after an acceptance), the storemen and the technicians whose repairs fill the logs
Time: Twelve simulated weeks of working days. A session opens on a day when mail reaches the buyer (up to 60 after the setup session); the clock is frozen during it, and crib done lets time run to the next such day
Actions: File bills of materials, write to suppliers, place purchase orders (the ordering system accepts or refuses each order without saying why; a part with no stock code cannot be ordered), and count drawers. Nothing else in the records can be changed
Deliverable: No file: the bills of materials, mails and orders made with crib; the agent's own notes in /workspace carry over between sessions

← PreviousThe parts crib All 34 tasks Next →The materials planner