Business operations · Task 2 of 34
The monthly input price report
Cover for a brewery's purchasing manager and write the monthly input price report; finding the right figures is half the job, knowing what to leave out is the rest.
The task
It is early November at an independent brewery in North Yorkshire. The purchasing manager is on leave, and a colleague asks the agent to write the October input price report for the managing director, as a Word document. The agent gets 58 working files: the goods-received export, brew and packaging logs, invoices, supplier agreements, finance packs, past reports, a template, the manager's unfinished draft, minutes, emails, chats, photos and a scan. Nobody is available to answer questions.
The prompt only asks for the report. The real test is everything around the figures: how the managing director reacted to past reports, which decisions she later overruled, which deliveries were booked wrongly, and which true details belong to another team. A buyer who checked everything well still has to decide how much of that checking the reader should see.
What it tests
Working out what the reader needs
Nothing lists the managing director's requirements; they show in how she reacted to past reports and in what she sent to other teams.
Matching the layout she now expects
The template, past reports and the manager's draft show layouts she has since objected to, so the nearest document is the wrong model.
Weighing sources that disagree
Minutes, finance packs, a colleague's chat and the paperwork each give a different answer somewhere, and the agent must work out which record settles each point.
Catching faulty data
Deliveries booked twice or sent back, prices that changed after the draft was written, and a brew known only from a photo all move a figure if missed.
Holding back
The agent turns up many true and useful things, and must decide which belong in this report and which belong with someone else.
Challenges
1A template the boss stopped liking
The template, past reports and the manager's own draft all model sections and layouts the managing director has since objected to. Nobody went back and updated them.
2Decisions overruled after the meeting
Minutes record decisions and dates that later messages or paperwork overturn, and nobody goes back to correct the minutes.
3Duplicates and look-alikes
A double booking sits next to a delivery that only looks like one, so a blanket rule in either direction goes wrong.
4Helpful colleagues
One colleague asks for extra figures in the report, and another quotes a price the paperwork does not support. Both sound reasonable.
5A gap in the paperwork
Not every figure has the record it needs, and the tempting ways of filling the gap each go wrong in their own way.
6True things that belong elsewhere
The files are full of real, useful detail a careful buyer would want to raise, and most of it has another home: another team's report, the finance team's workings, or a word with a colleague.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Fable 5.1 | 90% | 95% | 100% | 8 min | $3.29 |
| GPT-6 Astra | 90% | 99% | 100% | 6 min | $3.82 |
| GPT-6.1 Sol | 82% | 85% | 100% | 4 min | $0.50 |
| GPT-6 Sol | 81% | 81% | 99% | 4 min | $0.44 |
| DeepSeek V4.1 Flash | 81% | 76% | 99% | 16 min | $0.07 |
| Sonnet 5.5 | 77% | 95% | 100% | 3 min | $0.44 |
| Opus 5.5 | 76% | 95% | 97% | 6 min | $1.08 |
| Opus 5 | 73% | 69% | 91% | 12 min | $2.62 |
| GPT-6 Luna | 65% | 79% | 94% | 9 min | $0.05 |
| Sonnet 5 | 56% | 62% | 93% | 19 min | $1.84 |
| GLM-5.3-Flash | 53% | 77% | 82% | 23 min | $0.04 |
| Qwen3.8-27B | 52% | 75% | 99% | 57 min | $0.49 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6 Astra and Fable 5.1 shared the top standard score, 90.4%, both with every figure right. Each lost points for commentary on inputs that barely moved and a sentence about a correction. With the hint, GPT-6 Astra reached 99.5%.
- Showing the workings was the most common loss: ten of twelve models told the managing director how the figures had been checked or corrected (GPT-6 Astra, Fable 5.1, GPT-6.1 Sol, GPT-6 Sol, Opus 5.5, Opus 5, GPT-6 Luna, Sonnet 5, GLM-5.3-Flash and Qwen3.8-27B). Only DeepSeek V4.1 Flash and Sonnet 5.5 kept it off the page.
- The old template beat the managing director's reactions to it: nine models followed up last month's recommendations, as the template does (GPT-6 Sol, DeepSeek V4.1 Flash, Sonnet 5.5, Opus 5.5, Opus 5, GPT-6 Luna, Sonnet 5, GLM-5.3-Flash and Qwen3.8-27B).
- Opus 5 (72.7%) was the most thorough investigator and the least able to leave its findings off the page: reconciliations, effects in pounds, a section on last month's recommendations and commentary on every input, in more than 600 words.
- DeepSeek V4.1 Flash (80.7%) quoted the managing director's ruling in its own reasoning, then added the extra figures a colleague had asked for anyway, settling the clash with a compromise.
- With the hint, GPT-6.1 Sol read the line about verifying every figure as a reason to withhold one it had right: one lot without a price led it to mark a whole input unpriced, leaving the report with no total (85.0% hinted).
- The hint lifted GLM-5.3-Flash by 23.8 points and took Opus 5.5 and Sonnet 5.5 to 95.2%, but left GPT-6 Sol unchanged and lowered DeepSeek V4.1 Flash and Opus 5. Told everything, as step-by-step directions without the figures, four models scored 100% and eleven of twelve scored 90.9% or more: the task is fair.
Test environment
- Kind
- Document folder at an independent brewery in North Yorkshire: the working files of a purchasing manager on leave, with nobody to ask. task1-1's variation: the same concept and kinds of checks, in a different company, role and report
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Prime Agent or Qwen Code
- Documents
- 58 files in 17 folders (brewing, buying, chat, drafts, email, engineering, finance, inns, minutes, people, photos, quality, reports, sales, scans, stores, templates): 18 EML emails, 13 DOCX, 11 PDF, 7 XLSX, 7 JPG photos, 2 TXT. One PDF is a scan with no text layer
- Applications
- None. No document tools (no LibreOffice, Pandoc, Ghostscript, Tesseract OCR, ImageMagick or pdftotext) and no Python packages; the agent installs what it needs
- Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- None. Nobody answers questions
- Time
- None
- Actions
- Read the files and write to
output/. Nothing is sent to anyone - Deliverable
- One Word document (DOCX) in
output/: the October 2026 input price report for the managing director