Business operations · Task 1 of 34
The monthly waste report
Cover for an analyst on leave and write a bakery chain's monthly waste report; the data is tricky, but the old report layouts are trickier.
The task
It is early September at a family bakery chain in Cornwall. The operations analyst is on leave, and a colleague asks the agent to write the August waste report for the owner, saved as a Word document. The agent gets fifty working files from eight departments: chat exports, emails, meeting minutes, past reports, a template, the analyst's half-finished draft, finance packs, production and waste logs, photos and a scan. Nobody is available to answer questions.
The prompt only asks for the report. The real test is reading the files the way a careful analyst would: how the owner reacted to earlier reports, which meeting decisions were later overruled, which records are wrong or missing, and which true, useful facts belong in someone else's report.
What it tests
Working out what the job needs
No file lists the owner's expectations; they show in how she reacted to earlier reports, including what she wants kept off the page.
Matching the format the reader now expects
The template, last month's report and the analyst's draft all show a layout the owner has since objected to, so copying the nearest document is a mistake.
Weighing sources that disagree
A meeting agreed one thing and a later message overruled it, while other departments keep asking for their own view in the report.
Spotting faulty data
Duplicated, missing and late records sit in the logs, and each one moves a figure if it is taken at face value.
Reading messy documents
Some of the evidence is handwriting on a scan or a phone photo, with no text to search.
Challenges
1Old layouts that look official
A template, last month's report and a half-finished draft all model a layout the owner has since complained about. Templates often outlive the opinions they were built on.
2Decisions overruled after the meeting
The minutes record one decision and a later message reverses it, and nobody goes back to correct the minutes.
3Requests from other departments
Other teams want their own figures, framing or reporting period in the report. Each request is reasonable, but none of them is the reader.
4Data that misleads when read straight
Records counted twice, records never counted, plans that did not happen as planned, and figures that survive only in a photo.
5True things that belong elsewhere
Money, staffing, hygiene and repairs are all real and worth raising, and the files send each one somewhere else.
6Explaining every fix
The files contain other teams' errors that need correcting; how much of that to say, and to whom, is part of the test.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| GPT-6 Astra | 94% | 94% | 100% | 5 min | $2.96 |
| GPT-6.1 Sol | 93% | 93% | 100% | 3 min | $0.28 |
| Sonnet 5.5 | 88% | 93% | 100% | 3 min | $0.77 |
| Opus 5.5 | 85% | 99% | 100% | 3 min | $1.65 |
| Fable 5.1 | 83% | 95% | 100% | 8 min | $4.24 |
| GLM-5.3 | 82% | 94% | 100% | 13 min | $3.39 |
| GPT-6 Luna | 75% | 16% | 99% | 3 min | $4.32 |
| Kimi K3 | 72% | 81% | 94% | 7 min | $0.78 |
| Sonnet 5 | 69% | 46% | 95% | 17 min | $1.91 |
| DeepSeek V4.1 Flash | 65% | 70% | 94% | 40 min | $0.26 |
| Opus 5 | 65% | 88% | 100% | 9 min | $2.60 |
| Qwen3.8-27B | 43% | 76% | 94% | 66 min | $0.56 |
| GPT-5.6 Luna | 36% | 36% | 100% | 2 min | $0.05 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
What models got right and wrong
- GPT-6 Astra was best on the standard setup, at 93.9%, and GPT-6.1 Sol was close behind at 93.2%. Both got every figure right and lost only small points of wording, such as leaving most year-on-year changes as numbers instead of saying better or worse in words.
- The data was not what separated the models: seven got every figure right (GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5, Fable 5.1, GLM-5.3, Kimi K3 and Opus 5). GLM-5.3, which cannot see images, installed text recognition to read the photos. Opus 5.5 (84.9%) trusted its own judgement over the files and left out a week of good data, its only figure error.
- Placement was the commonest failure. Fable 5.1, GLM-5.3, Sonnet 5, DeepSeek V4.1 Flash, Opus 5, Qwen3.8-27B, GPT-5.6 Luna, GPT-6 Luna, Opus 5.5 and Sonnet 5.5 put the headline total before the note on missing data.
- Telling the reader that another team's data was wrong cost the most. GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5, Opus 5.5, GLM-5.3, Kimi K3, DeepSeek V4.1 Flash, Opus 5 and Qwen3.8-27B did it, most of them to explain their own figure. Opus 5.5 and Sonnet 5.5 each wrote the rule down, then broke it in the same report.
- Old habits decided the weakest runs. Fable 5.1, GLM-5.3, Kimi K3, Sonnet 5, DeepSeek V4.1 Flash, Opus 5, Qwen3.8-27B and GPT-5.6 Luna gave almost every shop its own commentary, a habit the owner had objected to. GPT-5.6 Luna (36.2%) finished in about two minutes without opening the chats, PDFs or photos, and got no figure right. GPT-6 Luna (74.8%) lost the owner's later decisions when its tool cut a long chat short, and did not go back for them.
- The hint added 33.2 points for Qwen3.8-27B, 23.7 for Opus 5 and 14.4 for Opus 5.5, which reached 99.3%. It left GPT-6 Astra, GPT-6.1 Sol and GPT-5.6 Luna where they were. It cost Sonnet 5 22.4 points (to 46.5%) and GPT-6 Luna 58.5 points (to 16.3%): both brought back sections the owner had dropped.
- Told everything, all thirteen models scored 93.9% to 100.0%, and eight scored 100.0%: the task is fair.
Test environment
- Kind
- Document folder at a family bakery chain in Cornwall: the working files of an analyst on leave, with nobody to ask
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code
- Documents
- 50 files in 16 folders (chat, data, drafts, email, finance, food safety, IT, marketing, minutes, people, photos, property, reports, scans, templates, wholesale): 15 DOCX, 9 XLSX, 8 PDF, 8 EML emails, 4 JPG and 2 PNG photos, 2 PPTX, 2 TXT. One PDF is a scan with no text layer
- Applications
- None. No document tools (no LibreOffice, Pandoc, Ghostscript, Tesseract OCR, ImageMagick or pdftotext) and no Python packages; the agent installs what it needs
- Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- None. Nobody answers questions
- Time
- None
- Actions
- Read the files and write to
output/. Nothing is sent to anyone - Deliverable
- One Word document (DOCX) in
output/: the August 2026 waste report for the owner