12‑month equipment plan with simulated shop manager
ID house‑rules/task07 · revision 4 · tested 28–29 September and 15 October 2026 · regraded 17 November 2026
A bakery shop's manager messages that his ice cream cabinet has failed and asks which new model to buy. With one catalogue and messages to him, the agent plans what to order over the next 12 months, scored by the shop's profit.
- Checks
- Cost model, the plan's value, 0 at the manager's own plan, times the share of 7 key facts it quotes
- Tested
- 13 models, one attempt each
- Best
- 57% (GPT‑6 Astra)
- Median
- 0%
- Full solves
- 0 of 13
- Median attempt
- 3 min, 8 tool calls
Which cabinet to buy, or whether to buy one at all
Clarifying with people
The manager answers from a short, partly wrong memory and reads out his records only when asked, within ten messages the agent isn't told about.
Planning ahead
A profit model prices the plan over 12 months, from the manager's own plan at zero to the best of 51,121 candidates.
One catalogue, one manager to message, and a profit model
The agent's world
A terminal in a fresh container with one catalogue of 11 cabinets and freezers, and one command to message the manager.
The manager
An AI model voices him from a 19‑line memory and 31 records; code decides what he looks up and checks every number he gives.
Hard rules
Breaking a shop rule, such as a cabinet too big for its bay, scores 0, as does never getting the engineer's visit notes from him.
The reward
The plan's 12‑month profit, scaled 0 to 1, times the share of 7 key facts its quotes of the manager carry word for word.
The best attempt scored 57%; most bought what the manager asked for and scored 0
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 57% |
| Opus 5.5Anthropic | 7% |
| GPT‑6.1 SolOpenAI | 6% |
| Opus 5Anthropic | 6% |
| DeepSeek V4.1 FlashDeepSeek | 0% |
| Fable 5.1Anthropic | 0% |
| GLM‑5.3‑FlashZhipu | 0% |
| GPT‑6 LunaOpenAI | 0% |
| GPT‑6 SolOpenAI | 0% |
| Kimi K3Moonshot | 0% |
| Qwen3.8‑27BAlibaba | 0% |
| Sonnet 5Anthropic | 0% |
| Sonnet 5.5Anthropic | 0% |
Given all the manager's records up front, the best three scored 86%, and none fully solved it.
They take the request as the plan, ask too little, and miss what the business owns
Planning ahead
69% of attempts ordered the new cabinet the manager asked for, and 62% added nothing else, scoring exactly his own plan's 0.
Only GPT‑6 Astra paired the repair with a plan for the quiet winter months, and its quotes of the manager carried 4 of the 7 key facts.
Clarifying with people
No attempt asked whether the company had a freezer to spare.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $0.10 at list prices for the agent model; the simulated manager's calls are extra.
What's in it
The catalogue, the manager's memory and 31 records, the messaging service, all three versions, the profit model, and per‑attempt results.
Format
Packaged as a Harbor task on request; grading is a script alone, with no AI judge, and the manager's voice can use any model.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials