Two‑site spares pick lists and stock plans
ID tb‑tasks/spares‑crib · revision 1 · results as of 16 November 2026
As stores controller for a coffee roaster's two sites, the agent keeps the technicians' list of parts for each repair, learned from a year of work logs, and a stock plan that keeps downtime risk under the company's limit. One session a week; only a workspace carries over.
- Checks
- 35, 33 repair reports and 2 stock checks
- Tested
- 6 models, one attempt each
- Best
- 100% (Opus 5.5)
- Median
- 90%
- Full solves
- 1 of 6
- Median attempt
- 22 min, 124 tool calls
Two sites' pick lists and stock, learned from a year of work logs
Long‑horizon work
Nine fresh sessions keep only the agent's files, and the stock plan, refiled week by week, is checked at weeks 4 and 8.
Stateful tool use
Stock moves as technicians take parts, and the stock plan is tested twice against the downtime limit at both sites.
Planning ahead
Stock plans must keep each site's downtime risk under the limit at weeks 4 and 8, within caps on orders and drawers.
Unstated requirements
Which parts a repair really takes shows only in the work logs; the documented lists are right on 12 of 33 reports.
Two sites' records in two formats, scripted technicians and nine sessions
The agent's world
A terminal in a container with one tool to a simulator: file the pick list and stock plan, count drawers, end the session.
The records
One site's scanned paper log as OCR text, with errors, and the other's database export; the two number parts differently.
The technicians
Played by code, they take the list's entry to the stores, reject what doesn't suit, use what the job needs and return the rest.
The reward
The pick list's accuracy on 33 repair reports, measured up from the manuals' 12 right, times the credit from two stock checks.
Three attempts scored 95% or more, and the two that failed a stock check 62% and 74%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 100% |
| Fable 5.1Anthropic | 95% |
| Sonnet 5.5Anthropic | 95% |
| GPT‑6 LunaOpenAI | 86% |
| GPT‑6 AstraOpenAI | 74% |
| GPT‑6.1 SolOpenAI | 62% |
They count the wrong part as cover, miss a new combined repair, and follow one‑offs
Stateful tool use
Two counted a shatterproof lamp as cover for the ordinary one; by week 8 neither site held an ordinary lamp, and the stock check failed.
Unstated requirements
67% picked the wrong parts for a two‑bearing failure at least once, though the work logs show it takes a shaft kit.
One dropped a belt's pins after a single job waited for them, and listed a manual's bearing it had seen technicians reject.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.11 to $12.33 for the agent model; nothing else in the environment calls a model.
What's in it
The simulator with both sites' records, the technicians' rules and failure schedule, the grader with its replay check, and a reference solution.
Format
Our attempts ran on its own multi‑step Harbor build, one step per session; other packaging on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The same plant's buying environment keeps the shelves stocked as suppliers withdraw parts. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials