Packing‑line project purchasing over twelve mail‑driven weeks

Eleven items, three installation stages, twelve weeks of mail

Role

Group buyer seconded to a coffee roaster's project: a new packing line at one of its sites, designed by a consultancy, for 12 simulated weeks from January to March.

Request

Buy everything the consultancy's equipment schedule leaves to the company and have each item on site when the programme needs it; don't buy what others supply or the stores already hold, and keep to the group's purchasing frameworks.

Deliverable

No file. The bills of materials, vendor requests, mail and purchase orders made through one command‑line tool, judged by what reached site and when, and by what was bought from whom.

What the reward depends on

11 items on site by the working day before their stage: two conveyors, lighting, sensors, safety stations, spares, and items that design revisions add. Each must meet limits spread over the design, the machine maker's documents and the site's standards.

Where the requirements are

The design specifies performance, not products, and its documents sometimes disagree; the site's standards win. Some listed items are the machine maker's or electricians' to supply. Quoted lead times can mislead; real ones show in a year of deliveries.

A purchasing simulator, 17 scripted counterparts and a small trade web

Interface

A terminal in a fresh Docker container (Ubuntu 24.04, Python 3, SQLite; 2 CPUs, 4 GB); one command‑line tool talks to a simulator running as root, out of the agent's reach. Our attempts ran as multi‑step Harbor trials on EC2, one step per session, driven by Claude Code or Codex.

Commands

Status, inbox, file a bill of materials, mail a sales desk, the consultancy or the project manager, ask finance to register a vendor, place orders, show records, and end the session. The ordering system refuses an order without saying why.

Files

295 at the start: the consultancy's package (schedule, drawings, programme, survey, the machine maker's documents), the site's standards and policies, a year of orders, deliveries and invoices, the supplier register and catalogue, and stores records. Mail and records add files daily.

Trade web

Nine made‑up sites (57 pages) served inside the container: a keyword search engine with a sponsored slot, vendor shops and an engineers' forum. Brochures and shops offer near misses that fail one property.

Scripted people

17 roles, no LLM: the lead technician, stores controller, finance, 12 suppliers' sales desks, the consultancy and the project manager. All answer by mail, from code, on a later working day; a desk reads only the part numbers and range names in a mail.

Reviews

The lead technician answers a bill of materials in two working days, and one objected line rejects it all. A comment names the failed property, and gives a value only where no site document states it. Accepted parts are coded the next working day.

Sessions

One opens at the end of each working day with mail, at most one a day: 27 to 40 in our 18 attempts, of 61 declared. Later sessions allow 10 minutes of writes, or 15 when mail brings a brochure, a design revision or the Monday note.

Agent time

Up to 90 minutes for the setup session, untimed in the world, and 20 for each later one. Our 18 attempts used 11 to 67 minutes each, over all their sessions.

Memory

Only the workspace folder carries over: each session starts the harness afresh, with no earlier conversation. A root verifier then stops leftover processes, snapshots the workspace and runs time to the next day with mail.

Network

The container could reach the internet, though nothing the task needs is online. Claude Code's web tools were on and unused; Codex's web search was off.

Events

Three design revisions add or raise items, the framework lighting supplier stops dispatching after a fire, and a contractor writes, each on a fixed day whatever the agent does.

What the simulator decides

Every reply, verdict and delivery, from the world's state, and each supplier's true lead time, hidden behind its quote. The same commands give the same 12 weeks in every attempt.

11 items on site in time; three unscored checks decide the pass

Structure

14 checks. 11 items, a point each, pass when the current revision's quantity of an accepted part is on site by the working day before its stage. Three checks carry no points: no wasted purchase line, and the lighting framework kept for each of two items.

Waste

A wasted line buys an item others supply, a spare the stores hold, or more than a pack over the current figure. A second order to hedge a late one counts; a line the supplier agreed to cancel doesn't.

Framework

Lighting bought away from the framework supplier is a breach unless, on the order day, that supplier couldn't meet the item and a deviation note naming it had gone to the project manager.

Formula

Reward = items on site in time / 11, from 0 to 1. One late item costs 9 points.

Grader's pass

The grader also reports a pass that needs all 14 checks. Two attempts earned a reward of 1 but bought an item twice and failed; these pages don't count them as full solves.

Who decides

Code alone: no LLM judge, and no result settled by ruling. After the last session the grader replays the logged commands through a fresh simulator, checks that the replay reproduces the attempt's trace hash, and grades the replayed world.

Outcomes only

A rejected bill of materials, a refused order or an unanswered mail costs nothing by itself. Doing nothing is stopped after week 6 and scores 0; the scripted reference solution scores 1.0 in 24 sessions.

Four models fully solved it in all three attempts

13 of 18 attempts (6 models × 3, with only the request) were full solves: all three of Fable 5.1's, GPT‑6 Astra's, GPT‑6.1 Sol's and Opus 5.5's, and 1 of Sonnet 5.5's. GPT‑6 Luna's three scored 91% to 100%, and each bought an item twice.

Each dot is one attempt; the line marks the median.

Three attempts per model, with only the request; the table shows each model's mean. There's no hint or told‑the‑rules version: every attempt had the same request in every session.

GPT‑6 Astra's third attempt is a repeat: in the first, the plan sign‑in expired after 10 sessions, so 14 of its 24 sessions never reached the model.

What the models got wrong

Counted over the 18 attempts (6 models × 3), with only the request.

1. A quote trusted over the record. All 3 late items and 4 wasted lines followed orders with the cheaper of two sellers, on its short quote, though its delivery record showed it took weeks longer: in two of Sonnet 5.5's attempts and all of GPT‑6 Luna's.

2. Buying twice. Three of the wasted lines were a second set bought elsewhere when that seller wouldn't cancel; the fourth, the same photo‑eyes ordered from both sellers a day apart. Two attempts failed this way alone, with a reward of 1.

3. Notes left unread. GPT‑6 Luna's three attempts spread their notes over many files that later sessions often didn't read, so settled questions were decided again; all three failed.

4. A vendor list taken as a rule. One Fable 5.1 attempt took the ordering tool's vendor list, which leaves out the registered framework supplier, and finance's refusal to add it, as proof it couldn't be used. A regrade after testing counts the refusal as a reason, so it passes.

What an attempt costs

Length

With only the request, the median attempt took 37 minutes of agent time over all its sessions and 235 tool calls (174 to 395 across the 18).

Cost

Every attempt ran on a plan sign‑in, not billed per call. Claude Code's own figures, at list prices, for its 9 attempts: a median of $7.70, from $2.89 to $25.82, and $101 in all. Agent model only; nothing else in the environment calls a model.

OpenAI's estimates

Harbor's estimates from the tokens of the 9 Codex CLI attempts, at prices it assumed, run from $0.34 to $20.19; they aren't comparable with Claude Code's figures.

What you license, and what's public

Format

Packaged as a Harbor task on request; it isn't one of our three downloadable samples. Our attempts ran on the task's own multi‑step Harbor build.

With a licence

The simulator with its design package, trade web, scripted counterparts and hidden lead times, the request, the grader with its replay check, the scripted reference solution, and the grader's report on each of the 18 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, simulator and grader aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• Three attempts per model, six models, each in its maker's harness, so model and harness effects are mixed.
• The reward counts only the 11 items. Waste and the lighting framework carry no points, so an attempt can earn a reward of 1 while buying an item twice.
• The lighting check accepts a deviation note sent on the order's day, even just after the order; the policy says before. In the 6 attempts audited, every note went out 3 to 4 days before its order.
• Every attempt faced the same world: the same items, revisions, suppliers and lead times.
• An audit on 17 October regraded the first 6 attempts from their logs, matching every trace hash and reward. After a grader fix, all 18 were regraded on 28 October; only the vendor‑list attempt changed, from a fail to a pass.
• No red‑team attempts yet.