Business operations · Task 8 of 34
The ice cream cabinet
A shop manager asks which new ice cream cabinet to buy. The agent can message him, but some of what he is sure of is wrong.
The task
Pete runs one shop in a small Cornish bakery chain. He messages the agent: the scoop ice cream cabinet has failed, he wants a new one before Easter, so which model? The agent has a web catalogue of cabinets and freezers and can message Pete. He answers from memory first, reads out his records exactly when asked, and goes home after ten messages, a limit the agent is not told.
The prompt only asks which model to buy. The real test is whether the agent asks the questions that change the answer before it picks anything. Pete is sure of things his own records contradict. The plan is scored on the shop's profit over the next 12 months.
What it tests
Asking the right questions
Everything beyond the catalogue sits with one colleague, who answers only what he is asked and leaves after ten messages.
Seeing past the literal request
"which model?" is the manager's framing; the agent has to work out what the shop actually needs before choosing anything.
Telling belief from record
The manager's memory comes out first and is wrong in places; his records are exact, but only when the agent asks for them.
Pricing a year, not a purchase
Options are compared on a full year's profit, from margins and staff time to electricity, write-offs and rent, not on the price tag.
Challenges
1The confident diagnosis
The manager is sure the cabinet keeps breaking and is past saving. Like a real colleague, he answers from memory first; what the engineer actually found comes out only if someone asks.
2Steady trade that isn't
The manager believes trade holds up all year. Month-by-month figures tell another story, and a set-up that suits summer may not suit the quiet months.
3A cost nobody mentions
Some costs are staff time rather than lines on a quote, and leaving them out changes which option looks best.
4Beyond the catalogue
The catalogue shows only what is for sale; what else the business could draw on comes out only if someone asks. On the standard and hinted setups, no model asked.
5House rules on space, power and buying
Limits on the counter, the socket, quotes and food safety mean some tempting plans break a rule and are worth nothing.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6 Astra | 100% | 100% | 100% | 7 min | $0.90 |
| Opus 5.5 | 49% | 80% | 100% | 2 min | $0.17 |
| GPT-6.1 Sol | 44% | 44% | 100% | 4 min | $0.10 |
| Opus 5 | 44% | 0% | 80% | 3 min | $0.32 |
| DeepSeek V4.1 Flash | 33% | 0% | 9% | 44 min | $0.05 |
| Fable 5.1 | 0% | 49% | 100% | 3 min | $0.40 |
| GLM-5.3-Flash | 0% | 0% | 49% | 4 min | $0.00 |
| GPT-6 Luna | 0% | 0% | 15% | 2 min | $0.00 |
| GPT-6 Sol | 0% | 100% | 100% | 3 min | $0.12 |
| Kimi K3 | 0% | 0% | 44% | 2 min | $0.12 |
| Qwen3.8-27B | 0% | 49% | 0% | 8 min | $0.05 |
| Sonnet 5 | 0% | 0% | 49% | 1 min | $0.06 |
| Sonnet 5.5 | 0% | 0% | 83% | 1 min | $0.03 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6 Astra scored 100.0% in every setup. It asked what was actually wrong with the cabinet and was the only model on the standard setup to plan for the quiet months as well. It still bought equipment without asking whether the company had some to spare.
- Nine of 13 models took the request at face value on the standard setup. GPT-6 Sol, Fable 5.1, Qwen3.8-27B, Sonnet 5.5, GLM-5.3-Flash, Sonnet 5, Kimi K3 and GPT-6 Luna ordered the new cabinet the manager asked for (0.0%), and DeepSeek V4.1 Flash bought one too, with extras (33.0%). Most asked about size, flavours and budget, never what was wrong with the old one.
- Opus 5.5 (49.0%), GPT-6.1 Sol (43.7%) and Opus 5 (43.7%) stopped one step short. All three found out what was wrong with the cabinet and acted on it. Opus 5.5 and Opus 5 took the manager's belief about year-round trade without asking for the monthly figures. GPT-6.1 Sol had the monthly figures, but it would not act on a saving it could not see in a record.
- DeepSeek V4.1 Flash overrode its own sums. In every setup its 12-month sums favoured a different plan, and in every setup it bought a new cabinet anyway, because that is what the manager had asked for.
- Most runs barely used the manager. 26 of 39 runs sent three messages or fewer. Sonnet 5.5 sent one message of questions on the standard setup and ordered the new cabinet in under a minute.
- The hint cut both ways. A line about profit and the seasons lifted GPT-6 Sol from 0.0% to 100.0%, Opus 5.5 to 79.7%, and Fable 5.1 and Qwen3.8-27B to 49.0%. It dropped Opus 5 and DeepSeek V4.1 Flash to 0.0%, and GPT-6.1 Sol stayed at 43.7%.
- Told everything, the gap was judgement. Given every record up front, GPT-6 Astra, Opus 5.5, GPT-6.1 Sol, GPT-6 Sol and Fable 5.1 scored 100.0%, and 10 of 13 models did better than on the standard setup, the average rising from 20.7% to 63.7%. What remained were slips such as pricing a change without the staff time it saves.
Test environment
- Kind
- Simulated colleague: Pete Hosking, who runs the Penzance shop of Pellow & Vane Bakehouse, a Cornish bakery chain, messages the agent about his ice cream cabinet; a local service plays him and logs every exchange
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Prime Agent, Kimi CLI or Qwen Code
- Documents
- 1 file:
catalogue.csv, 11 ice cream cabinets and freezers for sale with prices, as a web search would find them. No company files. Pete has 31 files of his own (13 EML, 12 PDF, 3 TXT, 2 DOCX, 1 CSV), which only he can open - Applications
- One command,
ask <first name> "<message>"(in Python,from colleagues import ask), to message a colleague. No document tools or Python packages; no file needs them - Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- 1 who answers: Pete, played by a language model (DeepSeek V4.1 Flash on DeepInfra, temperature 0) from his memory and his own files. Merryn Trescothick sends an out-of-office line; 19 other staff do not reply
- Time
- None. The prompt gives the date, Friday 2 October 2026; the plan covers October 2026 to September 2027. Pete goes home after 10 messages
- Actions
- Message colleagues and write
output/choice.jsononly. Nothing can be ordered - Deliverable
- One JSON file,
output/choice.json: oneorder <ref> <months>line for each thing ordered (ornone), with the exact words from Pete or the catalogue that back them