Packing‑line project purchasing over twelve mail‑driven weeks
ID tb‑tasks/line‑fitout · revision 1 · results as of 16 November 2026
As project buyer for a coffee roaster's new packing line, the agent turns a consultancy's design into accepted bills of materials and orders, so every item is on site before the stage that needs it. Sessions open on days with mail; only a workspace carries over.
- Checks
- 14, 11 items on site; each wasted line or breach of the supplier rules costs one
- Tested
- 6 models, 3 attempts each
- Best attempt
- 100% (5 models)
- Median
- 100%
- Full solves
- 13 of 18
- Median attempt
- 37 min
A packing line's parts bought over twelve weeks, each on site in time
Long‑horizon work
Dozens of sessions over twelve weeks keep only the agent's files, and orders placed weeks earlier decide what reaches site in time.
Stateful tool use
Orders placed over twelve weeks have to land before the installation step that needs them, though one seller's real lead times run long.
Planning ahead
Items, some added by design revisions, must reach site the working day before their stage, and quoted lead times can mislead.
A design package, a trade web, 17 scripted roles and a session per mail day
The agent's world
A terminal in a container with one command‑line tool to a simulator: file bills of materials, mail sellers, order and end the session.
The people
17 scripted roles answer by mail on a later working day; one objected line in a bill of materials rejects it all.
Sessions
One opens at the end of each working day with mail; only the workspace carries over, and events land on fixed days.
The reward
Items on site the working day before their stage, out of 11, less one per wasted line or unexplained buy outside agreed suppliers.
72% of attempts were full solves, all three of four models' among them
Each dot is one attempt, and the line marks the median.
| Model | Mean of 3 attempts |
|---|---|
| Fable 5.1Anthropic | 100% |
| GPT‑6 AstraOpenAI | 100% |
| GPT‑6.1 SolOpenAI | 100% |
| Opus 5.5Anthropic | 100% |
| Sonnet 5.5Anthropic | 94% |
| GPT‑6 LunaOpenAI | 85% |
They trust a short quote, buy twice, and lose track of their notes
Planning ahead
Every late item and every wasted line involved an order with the cheaper seller on its short quote, though its record showed it took weeks longer.
Stateful tool use
Three wasted lines were a second set bought elsewhere when a seller wouldn't cancel. Two attempts got every item on site in time and failed only for buying twice.
Long‑horizon work
GPT‑6 Luna spread its notes over many files that later sessions often didn't read, so settled questions were decided again.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.34 to $25.82 for the agent model; nothing else in the environment calls a model.
What's in it
The simulator with its design package, trade web, scripted people and hidden lead times, the grader with its replay check, and a scripted reference solution.
Format
Our attempts ran on its own multi‑step Harbor build, one step per session; other packaging on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials