AsapiThe AI Agent Trials
Asapi / Catalogue / Packing‑line project purchasing over twelve mail‑driven weeks
Environment datasheet · Manufacturing

Packing‑line project purchasing over twelve mail‑driven weeks

ID tb‑tasks/line‑fitout · revision 1 · results as of 16 November 2026

As project buyer for a coffee roaster's new packing line, the agent turns a consultancy's design into accepted bills of materials and orders, so every item is on site before the stage that needs it. Sessions open on days with mail; only a workspace carries over.

Checks
14, 11 items on site; each wasted line or breach of the supplier rules costs one
Tested
6 models, 3 attempts each
Best attempt
100% (5 models)
Median
100%
Full solves
13 of 18
Median attempt
37 min
What it tests

A packing line's parts bought over twelve weeks, each on site in time

Long‑horizon work

Dozens of sessions over twelve weeks keep only the agent's files, and orders placed weeks earlier decide what reaches site in time.

Stateful tool use

Orders placed over twelve weeks have to land before the installation step that needs them, though one seller's real lead times run long.

Planning ahead

Items, some added by design revisions, must reach site the working day before their stage, and quoted lead times can mislead.

How it works

A design package, a trade web, 17 scripted roles and a session per mail day

The agent's world

A terminal in a container with one command‑line tool to a simulator: file bills of materials, mail sellers, order and end the session.

The people

17 scripted roles answer by mail on a later working day; one objected line in a bill of materials rejects it all.

Sessions

One opens at the end of each working day with mail; only the workspace carries over, and events land on fixed days.

The reward

Items on site the working day before their stage, out of 11, less one per wasted line or unexplained buy outside agreed suppliers.

Results

72% of attempts were full solves, all three of four models' among them

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 models3 attempts each100%13 of 18
ModelMean of 3 attempts
Fable 5.1Anthropic100%
GPT‑6 AstraOpenAI100%
GPT‑6.1 SolOpenAI100%
Opus 5.5Anthropic100%
Sonnet 5.5Anthropic94%
GPT‑6 LunaOpenAI85%
What models get wrong

They trust a short quote, buy twice, and lose track of their notes

Planning ahead

  1. Every late item and every wasted line involved an order with the cheaper seller on its short quote, though its record showed it took weeks longer.

Stateful tool use

  1. Three wasted lines were a second set bought elsewhere when a seller wouldn't cancel. Two attempts got every item on site in time and failed only for buying twice.

Long‑horizon work

  1. GPT‑6 Luna spread its notes over many files that later sessions often didn't read, so settled questions were decided again.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.34 to $25.82 for the agent model; nothing else in the environment calls a model.

What's in it

The simulator with its design package, trade web, scripted people and hidden lead times, the grader with its replay check, and a scripted reference solution.

Format

Our attempts ran on its own multi‑step Harbor build, one step per session; other packaging on request.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.