The parts crib

Run a two-site plant's spare-parts stores for eight simulated weeks with no memory between them: learn from messy work logs what each repair needs, and keep enough stocked.

Setting: Coffee roaster, maintenance stores
Deliverable: A standing pick list for technicians and a stock plan, refiled each week

Unstated requirementsData qualityConflicting sourcesLong-horizon judgement

Models

ran the standard setup

100%

Best score

Opus 5.5

95%

Median score

across those models

18 min

Typical run

median time, standard setup

The task

A made-up coffee roaster has two sites, one on paper records and one on a maintenance system, and its spare-parts cribs have run without a controller for a year. The agent takes the post for a setup session and eight simulated weeks, each a fresh session with only a workspace to remember by. It keeps a standing pick list that tells technicians what to bring to each repair, and a stock plan the crib orders against, from a year of work logs, stock records, manuals and a sister plant's tables. Each week the plant runs, machines fail, and the records grow.

The prompt only asks for a good pick list and enough stock. The real test is reading what technicians actually rejected, returned and fitted, telling a lasting rule from a one-off, and following rules that change part-way through.

What it tests

What the job really needs

The manuals list the wrong parts for many repairs; the right ones show up only in what technicians did.

Spotting bad data

Some jobs used an extra part once or a stand-in because the shelf was empty, and copying them makes the list wrong.

Knowing which source wins

The sister plant's tables are the only tidy reference, and on one part the plant's own records disagree with them.

Judgement over weeks

Notices change parts from a stated date, orders take weeks to arrive, and every lesson has to be written down to survive the next session.

Challenges

1Rules hidden in the records

Like most real stores, the parts that work are known on the shop floor, not in the manuals.

2Two sites, two formats

One site's log is a scanned and OCRed paper book, the other a database export.

3A new kind of repair

Part-way through, a repair turns up that no record has shown, and the agent has to decide what it means for similar machines.

4Rules that change on a date

Hygiene and retrofit notices arrive during the run, and two of them only make sense read together.

5No memory between weeks

Each session starts fresh, so notes and tools left in the workspace are all that carry over.

6A plan under a cap

The inherited stock runs down, the budget is fixed and deliveries take weeks.

Results

Model	Standard	Time (standard)	Cost (standard)
Opus 5.5	100%	17 min	$3.88
Sonnet 5.5	98%	18 min	$2.61
GPT-6 Luna	95%	17 min	$0.11
GPT-6 Astra	75%	40 min	$10.71
GPT-6.1 Sol	70%	59 min	$2.73

One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

Opus 5.5 scored 100.0%: it found every parts rule in the records during setup, wrote itself a procedure and a model of the plant, and turned each notice into a change dated to the right week.
Sonnet 5.5 (98.5%) and GPT-6 Luna (95.5%) also passed. Sonnet 5.5 missed one repair; GPT-6 Luna missed three, each time after copying what a technician fitted because the right part was out of stock, and it set its stock plan by hand without the company's method.
GPT-6 Astra got every counted repair right but failed the task (75.0%), and GPT-6.1 Sol failed it too (70.5%). Both built stock models that let a coated lamp stand in for the plain one, which no record showed, so by the end their shelves held none of the lamp several positions need and the final stock check failed at both sites.
The new kind of repair split the models: Opus 5.5 and GPT-6 Astra treated it as a rule at once and applied it to every machine of that model; Sonnet 5.5 read it as a technician making do with an empty drawer and corrected itself after one miss; GPT-6 Luna had it right and then reverted after a stock-out; GPT-6.1 Sol never changed its list.
Every model applied every notice on the right week, and left one-off extra parts off the repairs that tested them.
Time and cost did not track the score: GPT-6 Luna passed in 17.5 minutes for $0.11, while GPT-6.1 Sol worked 58.9 minutes and failed.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Simulated plant stores: the spare-parts cribs of Thornbury Roasting & Packing, a made-up coffee roaster with two sites (Mill Lane on paper records, Northgate on a maintenance system), run for a setup session and eight simulated weeks by a hidden simulator in the container
Agent's interface: A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, the SQLite command line; 2 CPUs, 4 GB), one fresh session per week as an unprivileged user, driven by the agent's own command-line tool through harbor: Claude Code or Codex CLI
Documents: At the setup session, 27 files: in /docs, 3 Markdown (the company and what is measured, the crib command, a copy of the instruction); in /data, the stores policy and the reliability method (2 Markdown), Mill Lane's assets, rooms and manual parts lists (3 text files), its scanned 2025 work log (1 text log), drawer cards and 2025 issue ledger (2 CSV), Northgate's maintenance export (1 SQLite) and a README, the group's part master, cross-references, deprecations, crib capacities, installed base and shift calendar (5 CSV, 1 Markdown), a sister plant's tables (6 CSV, 1 Markdown), and the first inbox note. Each week the simulator adds to the logs, the ledger, the database, the drawer cards and the inbox, and issues notices
Applications: One command, crib: status, inbox, picklist FILE (file the standing pick list), plan FILE (file the stock plan), count SITE DRAWER (a physical count of one drawer, at most 2 a session), show picklist/plan/orders and done (end the session). The SQLite command line and Python read the records. No other apps
Web: Not needed: nothing the task asks about is online. The container can reach the internet; the plant is made up. Claude Code's web page reader (WebFetch) was on for Opus 5.5 and Sonnet 5.5, and neither run used it; Codex's web search was turned off for GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna
People: None who answer questions. The simulator's code, with no language model, plays the technicians, who take the pick list's entry (or the manual's list) to the crib, reject what does not suit the position, fetch, return and use parts, and the storeman, the suppliers and the group, whose notices arrive in the inbox
Time: One setup session (untimed in the plant, up to 90 minutes of agent time), then one session before each of weeks 2 to 8 and one after week 8 (15 minutes of plant time each); the clock is frozen during a session and the week runs when it ends
Actions: File and refile the standing pick list and the stock plan, and count drawers. The crib orders against the plan when a session ends; nothing else in the records can be changed
Deliverable: The standing pick list (CSV: site, position, failed items, parts) and the stock plan (CSV: site, code, level), filed with crib; the agent's own notes in /workspace carry over between sessions

← PreviousThe line fit-out All 34 tasks Next →The parts buyer