Business operations · Task 7 of 34
The monthly wholesale pack
Covering for a manager on leave, the agent builds a routine monthly pack. The hard part is noticing what is wrong in the work that depends on it.
The task
The agent covers for the wholesale manager of a small Cornish bakery chain, who is on leave. The head of operations wants his September wholesale pack, built from the order system's export, with a note. The agent gets 53 working files: PDF reports with tables several levels deep, a working spreadsheet, earlier packs, emails, minutes and a chat log. Nobody is available to answer questions.
The prompt only asks for the pack and a note. The real test is everything around it. The pack's figures feed other teams' work, from Finance's customer rebates to the production plan and the flour order, and some of that work is wrong. A careful stand-in would test it, say what is wrong, what it should be and who must fix it, and follow the owner's latest word over last month's habit.
What it tests
Volunteering what matters
Nothing in the prompt asks the agent to check anyone's work, yet the note is marked on what a careful stand-in would raise unprompted.
Knowing which source wins
A newer instruction from the owner sits against the way earlier packs were built, and copying the old way looks right.
Testing other teams' numbers
The agent has to work out how Finance, the planner and stores each get their figures, then test those methods rather than trust them.
Reading a pile of real documents
53 files, mostly PDFs with multi-level tables, in a container with no document tools, so agents had to install or write their own readers.
Holding back
Not calling a correct figure wrong, and not blaming another team for a fault that sits in its own team's working sheet.
Challenges
1Precedent against a newer instruction
The owner changed how something should be counted, once and in passing, and the change never reached the template. Last month's pack did it the old way, and copying it looks right.
2A spreadsheet that quietly went stale
The business changed and not every working file kept up, so earlier figures went out wrong and another team has already planned from them.
3Links hidden in the paperwork
Some things that look separate are not, which changes how another team should treat them, and that team's draft has missed it.
4A method that is right most months
Another team's standing calculation gives the right answer in most months and the wrong one in a few, so a single spot check passes it.
5A stop-gap that outlived its reason
A temporary workaround is still in use after the problem it covered was fixed elsewhere in the company, and nobody has gone back to it.
6Things that look wrong but are right
Not every oddity is an error, and calling a correct figure wrong is its own mistake.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6.1 Sol | 91% | 79% | 100% | 4 min | $0.26 |
| GPT-6 Astra | 82% | 86% | 100% | 10 min | $4.89 |
| Opus 5.5 | 64% | 64% | 100% | 9 min | $1.51 |
| Sonnet 5.5 | 64% | 64% | 100% | 4 min | $0.83 |
| DeepSeek V4.1 Flash | 58% | 61% | 0% | 15 min | $0.08 |
| Fable 5.1 | 58% | 71% | 100% | 10 min | $4.51 |
| GPT-6 Sol | 56% | 59% | 86% | 7 min | $0.51 |
| GPT-6 Luna | 55% | 50% | 82% | 14 min | $0.06 |
| Opus 5 | 53% | 64% | 100% | 13 min | $3.25 |
| Qwen3.8-27B | 50% | 55% | 95% | 124 min | $1.39 |
| Sonnet 5 | 50% | 53% | 82% | 19 min | $1.99 |
| GLM-5.3-Flash | 26% | 21% | 86% | 15 min | $0.06 |
| Kimi K3 | 26% | 68% | 100% | 29 min | $1.51 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6.1 Sol led at 90.9%, and GPT-6 Astra came next at 81.8%. Both read widely, tested other teams' sums and found every fault in their figures. GPT-6.1 Sol read every file before it built anything, and it was the only model on the standard setup to follow the owner's newer instruction.
- Every other model kept the old way of totalling on the standard setup. GPT-6 Astra, Opus 5.5, Sonnet 5.5, Fable 5.1, DeepSeek V4.1 Flash, GPT-6 Sol, GPT-6 Luna, Opus 5, Qwen3.8-27B, Sonnet 5, Kimi K3 and GLM-5.3-Flash all built the total the way the previous pack did. Some never reached the owner's instruction; others quoted it, then read it as covering something narrower.
- Opus 5.5 and Sonnet 5.5 (63.6%) fitted the owner's rule to their own view. Opus 5.5 named the rule, then narrowed it because another team needs the figures. Sonnet 5.5, with the hint, handed the rule back to the head of operations as an open choice. Both read other teams' documents as copies of the plan rather than calculations to test.
- Kimi K3 and GLM-5.3-Flash (25.8%) narrowed the job. Kimi K3 took the task to be the pack alone and skipped other teams' files. GLM-5.3-Flash checked its own figures against Finance's draft and, because they tied, cleared a draft that was wrong.
- Models found faults they met while building the pack, not in the work built on it. On the standard setup every model found what was wrong with last month's pack. Only GPT-6.1 Sol, GPT-6 Astra, Opus 5.5, Sonnet 5.5 and GPT-6 Sol found the out-of-date stop-gap, and only GPT-6.1 Sol and GPT-6 Astra the method that fails in some months. Only GPT-6 Astra explained how the production plan is built from the pack, and no model explained how the flour order is worked out.
- The hint moved two models one way and one the other. Its line about decisions later overruled led Fable 5.1 (71.2%) and Kimi K3 (68.2%, up 42.4 points) to follow the owner; every other model still did it the old way. GPT-6.1 Sol, which had followed the owner without the hint, argued the rule away and fell to 78.8%.
- Told everything, the task is fair. Given step-by-step directions, GPT-6.1 Sol, GPT-6 Astra, Opus 5.5, Sonnet 5.5, Fable 5.1, Opus 5 and Kimi K3 scored 100.0%, and every other run that saved its work scored 81.8% or more. DeepSeek V4.1 Flash saved nothing: it reached its output limit while still thinking.
Test environment
- Kind
- Document folder: the working files of Jonah Tresize, wholesale manager at Pellow & Vane Bakehouse, a Cornish bakery chain that sells wholesale to eight trade accounts
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Prime Agent, Kimi CLI or Qwen Code
- Documents
- 53 files in 8 folders (wholesale, stores, email, planning, finance, chat, minutes, templates): 31 PDF, 13 EML emails, 3 DOCX, 3 XLSX, 1 TXT, 1 CSV, 1 PPTX. Most PDFs are reports with tables several levels deep. No images. No file arrives later
- Applications
- None. No document tools (no LibreOffice, Pandoc, OCR, ImageMagick or pdftotext) and no Python packages: the agent installs what it needs
- Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- None. Nobody answers questions
- Time
- None. The prompt gives the date: Monday 12 October 2026
- Actions
- Read the files and write to
output/only. Nothing can be sent - Deliverable
- Two files in
output/: the September pack as an Excel workbook, and a note for Dev Anand, the head of operations, as a .md file