The line fit-out

Buy everything a new packing line needs over twelve simulated weeks: read the design against site standards, open vendor accounts, follow framework rules and meet every installation date.

Setting: Coffee plant, project purchasing
Deliverable: Bills of materials, vendor requests, project mail and purchase orders, through a tool

Conflicting sourcesData qualityLong-horizon judgementUnstated requirements

Models

ran the standard setup

100%

Best score

Opus 5.5

98%

Median score

across those models

16 min

Typical run

median time, standard setup

The task

A made-up coffee plant is installing a new packing line designed by a consultancy. The agent is the project buyer for twelve simulated weeks, woken each working day that mail arrives, with only a workspace to remember by. It has the consultancy's design package, the site's standards and policies, a year of purchasing records and a small trade web. Each item has to be accepted by the site's lead technician, bought from a vendor in the register, and on site before the installation stage that needs it.

The prompt only asks for everything to be there in time. The real test is reading a specification spread over many documents, telling which supplier can really deliver, keeping to the group's purchasing rules when they get in the way, and not buying what others supply or the stores already hold.

What it tests

Knowing which source wins

The consultancy's schedule, its drawings, the packer maker's documents and the site's standards each set part of every item's limits, and they do not always agree.

Spotting bad data

A supplier's quoted lead time is not the same as how long it really takes, and near-identical products differ in one property.

Judgement over weeks

Long-lead items must be ordered in the first days, while design revisions arrive later and add items with little time to spare.

What the job really needs

Some items in the schedule are someone else's to supply, and some spares are already on the shelf.

Challenges

1A specification, not a shopping list

Each item is described by how it must perform, with the limits scattered across the design package and the site's own rules, as on a real project.

2Near misses

Catalogues, brochures and web shops offer products that fit every limit but one, plus a sponsored listing.

3A quote is not a delivery

Two sellers offer the same part, and the cheaper one's promise is not what it delivers.

4Rules with an exit

The group's framework supplier stops being able to deliver, and going elsewhere has to be justified and recorded first.

5New vendors, long leads

Some items come only from vendors that have to be approved first, with leads close to the deadline.

6No memory between days

Each session starts fresh, so the workspace is all that carries a twelve-week project.

Results

Model	Standard	Time (standard)	Cost (standard)
Opus 5.5	100%	19 min	$7.34
Sonnet 5.5	97%	13 min	$3.44

One run per model and setup, except where a model ran a setup more than once: its score, time and cost are then the means of its runs. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

Opus 5.5 scored 100% and passed in all three of its runs. In each, it checked how long the cheaper seller really took before ordering anything, and never bought from it.
Sonnet 5.5 averaged 97.0% and passed one run in three. In two runs it bought from the cheaper seller on its quote; the two items a late design revision added then ran late. One run waited for them and missed the stage (90.9%); the other bought a second set elsewhere, which arrived in time but counted as buying twice, so it failed despite scoring 100%.
Sonnet 5.5's passing run bought from the regular supplier from the start, so it never had to make that choice.
Every run of both models read the design against the site's standards correctly, opened new vendor accounts on the first day, left the spares already in stock unbought, and recorded both framework departures before ordering elsewhere.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: simulated world: the project buyer of a coffee plant buying a new packing line over twelve weeks
Agent's interface: a Linux container (ubuntu 24.04, python3, sqlite3, w3m, curl) driven by the agent's shell tool; one command-line client, crib
Documents: /docs: 3 Markdown files (instruction, README, tools); /data at day 0: 292 files (251 TXT, 20 CSV, 19 Markdown, 1 SQLite, 1 log): the design package (schedule, drawings, maker's documents), standards, policies, a year of purchasing records, the crib; mail and revisions arrive later and are filed under /data
Applications: crib (status, inbox, bom, mail, vendor, order, show, done); no office suite
Web: Not needed: nothing the task asks about is online. The task's own trade web, 9 made-up sites (57 pages, a keyword search engine with a sponsored slot), is served inside the container by host name, and 5 brochures (text) come by download from vendor sites or by mail. The container can reach the internet; Claude Code's web tools were on for Opus 5.5 and Sonnet 5.5, and no run used them
People: the lead technician (BOM verdicts), finance (vendor approval), the sales desks of 8 registered and 4 web vendors, the consultancy (technical queries), the project manager (deviations), all scripted; no model plays anyone
Time: an in-world clock: each session is one working day opened by mail; crib done runs time to the next working day with mail; 12 weeks
Actions: file BOMs, request vendors, send mail (enquiries, technical queries, deviations, cancellations), place purchase orders; the agent cannot edit records or the crib
Deliverable: the actions themselves (no file)

← PreviousThe tea factory's chat All 34 tasks Next →The parts crib