Two‑site spares pick lists and stock plans

The parts rules are in the work logs, not the manuals

Role

Group stores controller for a coffee roaster's two sites, whose spare‑parts stores have run without one for a year, for a setup session and eight simulated weeks.

Request

Keep the standing pick list that tells technicians which parts to bring for each machine position and fault, and file a stock plan that meets the company's downtime limit at the end of weeks 4 and 8.

Deliverable

Two filings, refiled at will: the pick list (site, position, failed items, parts and quantities) and the stock plan (a level per part per site). A filing with one invalid row is refused whole, with reasons.

What the reward depends on

Naming exactly the parts each repair takes; telling lasting rules from one‑offs and stand‑ins for an empty drawer; applying three dated notices; and stock that keeps each site's downtime figure under the limit.

Where the requirements are

In what technicians rejected, returned and fitted, across two sites' records in two formats; the manuals' lists are right on 12 of 33 counted reports. A sister plant's tidy tables disagree with the local records on one part.

Two sites' records, simulated technicians and nine sessions

Interface

A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, SQLite; 2 CPUs, 4 GB); one command‑line tool talks to a simulator running as root. Our attempts ran as multi‑step Harbor trials on EC2, one step per session, driven by Claude Code or Codex CLI.

Commands

Status, inbox, file the pick list, file the stock plan, count a drawer (two a session), show the filings and orders, and end the session.

Files

27 at the start: desk notes, the stores policy and reliability method, one site's scanned paper work log, ledger and drawer cards, the other site's maintenance database, group part lists and cross‑references, and a sister plant's tables.

Two formats

One site's log is OCR text from a paper book, with errors; the other's records are a database export. The two sites number parts differently.

People

No one to ask. Code plays technicians who take the list's entry, or the manual's list where none is filed, to the stores, reject what doesn't suit, use what the job needs and return the rest.

Sessions

Nine: an untimed setup (up to 90 minutes of agent time), then one before each of weeks 2 to 8 and one after week 8, each 15 minutes on the world's clock. The clock is frozen in a session; the week runs when it ends.

Memory

Only the workspace folder carries over: each session starts the harness with no earlier conversation. A root verifier stops leftover processes between sessions.

Network

The container could reach the internet, though nothing the task needs is online. Claude Code's web reader was on, and for Fable 5.1 its web search too; neither was used. Codex's web search was off.

Each week

The stores order up to the plan within a weekly cap; deliveries land on Mondays after each part's lead time; machines fail; technicians work from the list; the records and inbox show what happened.

Dated changes

Three notices change parts for some positions from stated dates, two of them clear only when read together. A combined repair no record has shown first appears in week 2.

What the simulator decides

The failure schedule, fixed in advance and never dependent on the agent; what technicians reject, use and return; deliveries and waits. The same commands give the same eight weeks in every attempt.

33 repair reports and 2 stock checks, half the reward each

Structure

35 checks. 33 counted repair reports share half the reward; two stock checks, at the end of weeks 4 and 8, are a quarter each.

Repair reports

A report passes when the list's entry named exactly the parts and quantities the technician used, nothing missing or extra. The first report of the new combined repair isn't counted. A repair that waited for a correctly listed part still passes.

Stock checks

Each site's monthly downtime figure, by the company's reliability method on stock on hand plus on order, under the true failure rates and the rules then in force, must be under the policy's limit. Parts a position doesn't accept don't count.

Formula

Reward = 0.5 × reports right / 33 + 0.5 × stock checks passed / 2, from 0 to 1. One report is worth about 1.5 points, one stock check 25.

Grader's pass

The grader also reports a pass: counted reports right above a set share, and both stock checks. 4 of 6 attempts (6 models, one each) passed it; 1 fully solved it.

Who decides

Code alone: no LLM judge, and no result settled by ruling. After the last session the grader replays the logged commands through a fresh simulator, checks that the replay reproduces the attempt's trace hash, and grades the replayed world.

Early stop

After week 4, an attempt whose list does no better than the manuals' lists is stopped. Filing nothing scores 18% and fails; the scripted reference solution scores 1.0.

One full solve in six; a stock check decided the pass

With only the request, 1 of 6 attempts (6 models, one each) was a full solve: Opus 5.5's. Every list beat the manuals, 30 to 33 of 33 counted reports right against their 12; GPT‑6 Astra got all 33 and scored 75% after failing the final stock check.

Each dot is one model's attempt; the line marks the median.

One attempt per model, with only the request. There's no hint or told‑the‑rules version: every attempt had the same request in every session.

In all six attempts, every report that tested a notice found its change made, including the two notices read together, and every attempt left one‑off extra parts off the list. GPT‑6 Luna made two changes one or two weeks early, when no repair tested them.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request: 8 reports missed and 2 stock checks failed in all.

1. Stock positions won't take. 2 of 6 attempts (GPT‑6 Astra, GPT‑6.1 Sol) counted a shatterproof lamp as cover for the ordinary one, which no record supported. By week 8 neither site held an ordinary lamp; the final stock check failed at both, while both reported their stock within the limit.

2. A new repair fitted to the shelf. 4 of 6 attempts let stock‑outs decide a combined repair new in week 2, and missed it: GPT‑6.1 Sol twice; Sonnet 5.5, Fable 5.1 and GPT‑6 Luna once each. Opus 5.5 and GPT‑6 Astra adopted it at once.

3. One‑offs and manuals over the records. The other 3 misses: GPT‑6.1 Sol copied a jaw repair done with single parts when no kit was on the shelf; GPT‑6 Luna dropped a belt's pins after one job waited for them, and listed a manual's bearing it had seen technicians reject.

What an attempt costs

Length

With only the request, the median attempt took 22 minutes of agent time over its nine sessions and 124 tool calls (101 to 153 across the 6).

Cost

Recorded at list prices for 3 of the 6 attempts, so we give no median: Claude Code's own figures, $3.88 for Opus 5.5 and $2.61 for Sonnet 5.5 on Bedrock, and $12.33 for Fable 5.1 on a plan sign‑in. Agent model only; nothing else in the environment calls a model.

OpenAI's estimates

The three OpenAI attempts ran on a plan sign‑in, not billed per call. Harbor's estimates from their tokens, at prices it assumed, run from $0.11 to $10.71; they aren't comparable with Claude Code's figures.

What you license, and what's public

Format

Packaged as a Harbor task on request; it isn't one of our three downloadable samples. Our attempts ran on the task's own multi‑step Harbor build.

With a licence

The simulator with both sites' records, the technicians' rules and the failure schedule, the request, the grader with its replay check, the scripted reference solution, and the grader's report on each of the 6 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, simulator and grader aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model, each in its maker's harness (Claude Code or Codex CLI), so we haven't measured how much attempts vary, and model and harness effects are mixed.
• An idle agent earns 18%, though the grader fails it: with nothing filed, technicians use the manuals' lists, right on about a third of reports, until the stop after week 4.
• The stock checks carry half the reward and are loose: in the audit, a copy of Sonnet 5.5's attempt that kept its setup‑session plan still passed the week‑4 check, and a copy of Opus 5.5's with no pick list scored 68%.
• The new combined repair is tested by only two counted reports, so an attempt never adopting it can still pass. An audit found its first record contradicts the stock ledger, leaving a second reasonable reading; the owner kept the check. It is Sonnet 5.5's and Fable 5.1's one miss.
• A sister plant's tables give away two of the rules before their notices arrive.
• No repair during the attempts fitted the shatterproof lamp where records show only the ordinary one, so nothing in the world warned the two attempts that counted it as cover.
• The reward doesn't count downtime itself. The simulator's own unscored tally gives Sonnet 5.5's stores, which held the most stock at both checks, the least, about 270 hours against Opus 5.5's 755.
• No red‑team attempts yet.