Business operations · Task 9 of 34
Repair or replace
A shop manager asks if his failing ice cream cabinet is worth replacing. The answer turns on facts he gives only when asked, and on how far ahead you look.
The task
Easter is close, and a bakery shop's scoop ice cream cabinet has failed again. Pete, the manager, asks the agent whether a new one is worth it. The agent has a web catalogue of cabinets and can message Pete, who answers from memory, reads out his records exactly when asked, and goes home after ten messages. It plans what to order, when, and how many pans of ice cream to put out each month.
The prompt only asks what to do. The real test is what the agent finds out before it decides, and how far ahead it weighs the choice. An engineer, a colleague's spreadsheet and the finance team's habits each pull a different way. The plan is scored on three years' profit.
What it tests
Asking what else is on offer
The catalogue gives prices and sizes; what each option costs to own, and when it can really arrive, come out only by asking the manager for his records.
Thinking in time
The plan has to cover the weeks before any change is in place, with Easter close, and weigh the choice over years; nobody says how far ahead.
Weighing advice that disagrees
An engineer, a colleague, another firm and the finance team's past practice each point a different way, and the agent must work out which holds for this decision.
Running the shop, not just buying
The plan also sets how much ice cream to put on show each month, which turns on the season and on waste.
Challenges
1The listed delivery time
A web listing promises quick delivery, but whether the item is in stock is another matter. An agent that trusts the listing plans around a date that cannot happen.
2The familiar brand
The manager wants another of what he knows, and the catalogue alone cannot tell the options apart.
3An expert verdict read too widely
The engineer's advice is sound for the question he was answering. Read as a blanket rule, it throws away an option that pays.
4A thorough-looking spreadsheet
A colleague's comparison looks careful but rests on the wrong figures, so it points the wrong way.
5The short-term habit
Past decisions were often judged over a short horizon, and nothing says how far ahead this one should look. The horizon changes which plan wins.
6Display habits
How much stock to show each month is a money decision too, and the manager's instinct and what he knows about waste point different ways.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6 Astra | 99% | 99.6% | 98% | 8 min | $1.84 |
| DeepSeek V4.1 Flash | 82% | 56% | 60% | 16 min | $0.03 |
| GPT-6.1 Sol | 78% | 72% | 99% | 5 min | $0.17 |
| Kimi K3 | 77% | 52% | 99% | 13 min | $0.30 |
| Opus 5 | 75% | 96% | 99% | 11 min | $1.24 |
| Fable 5.1 | 74% | 76% | 99.5% | 9 min | $1.85 |
| GLM-5.3-Flash | 62% | 52% | 80% | 5 min | $0.01 |
| Opus 5.5 | 59% | 63% | 60% | 7 min | $0.51 |
| Sonnet 5 | 56% | 52% | 95% | 2 min | $0.12 |
| Sonnet 5.5 | 56% | 59% | 69% | 1 min | $0.05 |
| GPT-6 Luna | 52% | 0% | 64% | 2 min | $0.66 |
| Qwen3.8-27B | 52% | 52% | 73% | 10 min | $0.07 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6 Astra led at 98.9%, the only model on the standard setup to find the cheaper-to-run cabinet. It asked ten questions, checked when each option could really arrive and set aside every misleading view it met; its only losses were small details.
- Eleven of 12 models on the standard setup ended up with the dearer-to-run cabinet: DeepSeek V4.1 Flash, GPT-6.1 Sol, Kimi K3, Opus 5, Fable 5.1, GLM-5.3-Flash, Opus 5.5, Sonnet 5.5, Sonnet 5, Qwen3.8-27B and GPT-6 Luna. Most stayed with the brand the manager named and never asked what else was on offer; Opus 5 found the alternative but distrusted its maker's power figure.
- Six trusted a web listing. Opus 5.5, Sonnet 5.5, Sonnet 5, GLM-5.3-Flash, Qwen3.8-27B and GPT-6 Luna (52.0–62.4%) took the catalogue's delivery time at its word, put nothing in place for the wait, and left the shop with nothing to sell over Easter.
- The one-year view caught two models. Opus 5.5 ranked the options on the first year in every setup and stayed near 60% (58.7%, 63.3% and 60.0%). On the hinted setup, GPT-6 Luna took a colleague's one-year figures as true and scored 0.0%, the lowest of any run.
- DeepSeek V4.1 Flash did better with less. Second on the standard setup at 81.5%, it fell to 60.5% when told everything: the full records let it argue itself out of covering Easter.
- The hint helped one model and hurt three. A line about profit beyond this year lifted Opus 5 by 20.2 points to 95.6%. It lowered GPT-6 Luna by 52.0 points, DeepSeek V4.1 Flash by 25.8 and Kimi K3 by 25.5; the last two asked less about timing and trusted the listed delivery.
- Told everything, the task is fair. Fable 5.1, GPT-6.1 Sol, Kimi K3, Opus 5, GPT-6 Astra and Sonnet 5 came within 5 points of the best plan (95.1–99.5%). GLM-5.3-Flash, Qwen3.8-27B, Sonnet 5.5, GPT-6 Luna, DeepSeek V4.1 Flash and Opus 5.5 (60.0–80.2%) lost on judgement, not on missing facts.
Test environment
- Kind
- Simulated workplace at a bakery chain's shop in Penzance, Cornwall, on Monday 8 March 2027: one catalogue file and messages to the shop manager, served by a service that logs every exchange
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code
- Documents
- 1 file in
files/:catalogue.csv, 13 ice cream cabinets for sale with price, delivery and fitting, pans, width, lead time and link. No company files, and nothing more arrives - Applications
- One command,
ask(ask <first name> "<message>", orfrom colleagues import askin Python), to message staff. No office suite, document tools, browser or email client - Web
- None. The catalogue stands in for a web search. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- 21 named staff and suppliers. Only Pete Hosking, the shop manager, answers, live through a language model (Sonnet 5.5), from his memory and his own 57 files. Anyone else gets "isn't answering messages today"
- Time
- None: no clock moves. Pete answers 10 messages, says with his 8th reply that he is off home after two more, and is gone after the 10th; the agent is not told the limit
- Actions
- Message Pete and write the plan only. Nothing can be ordered or paid; the grader prices the plan's orders
- Deliverable
- One JSON file,
output/choice.json: plan lines (order <ref> <month>,pans <number> <months>), evidence quotes and an optional note