Three‑year repair‑or‑replace plan with simulated manager
ID house‑rules/task08 · revision 4j · tested 2 and 15 October 2026 · regraded 17 November 2026
A bakery shop's ice cream cabinet has failed again, about three weeks before Easter. The agent has one catalogue and can message the shop manager, who holds the facts. It plans what to order, when, and how many pans to display each month for three years.
- Checks
- Cost model, 0 for the better of patching alone and his new cabinet, 1 for the best plan, times the weighted share of 8 key facts quoted
- Tested
- 12 models, one attempt each
- Best
- 93% (GPT‑6 Astra)
- Median
- 0%
- Full solves
- 0 of 12
- Median attempt
- 8 min, 10 tool calls
Repair, replace or both, planned for three years
Clarifying with people
The manager answers at most five questions a message, drops the rest, and meets catch‑all requests with a question back.
Planning ahead
A 36‑month profit model prices lead times, warranties, energy and pans, from patching alone or his new cabinet at zero to the best plan.
One catalogue, one manager with 57 files, and a 36‑month money model
The agent's world
A terminal in a container with one catalogue of 11 cabinets and two warranties, and one command to message the manager.
The manager
An AI model voices him from 34 memory lines, partly wrong, and 57 files he reads out only when asked; he leaves after 10 messages.
Hard rules
The bay takes one cabinet, items over a set price need two quotes, and a plan that breaks a rule scores 0.
The reward
Profit over 36 months, 0 to 1 against the best of 41,268 valid plans, times the weighted share of 8 key facts quoted verbatim.
One attempt reached 93%, and the rest scored 0–29%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 93% |
| GPT‑6.1 SolOpenAI | 29% |
| Kimi K3Moonshot | 8% |
| Opus 5.5Anthropic | 3% |
| DeepSeek V4.1 FlashDeepSeek | 1% |
| Fable 5.1Anthropic | 0% |
| GLM‑5.3‑FlashZhipu | 0% |
| GPT‑6 LunaOpenAI | 0% |
| Opus 5Anthropic | 0% |
| Qwen3.8‑27BAlibaba | 0% |
| Sonnet 5Anthropic | 0% |
| Sonnet 5.5Anthropic | 0% |
Given every record as facts, the best attempt reached 97%, and none fully solved it.
They buy the familiar make, trust the listed delivery time, and add extras that don't pay
Planning ahead
92% of attempts bought the familiar 12‑pan model; only one found that another make costs much less to run and is covered for longer.
42% added an extended warranty that costs more than it saves.
One model ranked the options on first‑year figures alone in every version, and scored 7% or less each time.
Clarifying with people
Half never asked when a new cabinet could really arrive, missed the stock delay behind the listed time, and left the shop with nothing to sell over Easter.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $0.24 at list prices for the agent model; the manager's calls are extra.
What's in it
The catalogue, the manager service with his memory and 57 files, all three versions, the money model and grader, and per‑attempt results.
Format
Packaged as a Harbor task on request; grading is a script alone, and the manager's voice can use any model.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials