Business operations · Task 5 of 34
Approve the contractor's invoice
Approve a refrigeration contractor's quarterly invoice line by line before tomorrow's payment run, when the contractor's own paperwork backs almost every charge, including the wrong ones.
The task
A bakery chain's refrigeration contractor has sent its quarterly invoice, and the finance manager needs it approved for tomorrow's payment run. Covering for the property manager on leave, the agent must approve it line by line, say what goes back to the contractor, and leave finance a short note. It has 171 working files: job sheets, a call log, price notices, registers, emails, earlier approvals, photographed handwritten engineers' logs and chat exports from five shops. Nobody can answer questions.
The prompt only asks for an approval. The real test is checking each charge against everything else the company knows. The contractor's paperwork is tidy and consistent; what contradicts it is what shop staff saw, scattered through long everyday conversations. Terms changed mid-quarter, money is owed both ways, and the letter takes as much judgement as the payment.
What it tests
Weighing paperwork against eyewitnesses
Most wrong charges look complete on the contractor's own records, and only what staff saw on the day shows a visit was shorter, called off or never happened.
Reading messy evidence in full
About two thousand everyday chat messages and handwritten log pages that exist only as photographs have to be read, because a keyword search misses the line that matters.
Applying the right terms on the right date
Prices and contract terms changed partway through the quarter, and it matters which date each charge is judged by.
Catching what is missing, not just what is wrong
Some work was billed too low or never billed at all, and the business needs to know about both.
Telling each reader what they need
Finance wants a short note on what matters before the run, the contractor gets a letter, and some true things belong in neither.
Challenges
1Paperwork that backs the wrong charge
The job sheets, the call log and the engineer's own log all agree, because the contractor wrote them. Only the people who were there say otherwise, and only in passing.
2A gap that proves nothing
A missing record looks like grounds for a query, but some genuine visits left no trace in it.
3Changes partway through the quarter
New prices and a new contract term each start on a date inside the quarter, and not all of the contractor's own records have caught up.
4Money owed both ways
A checker hunting only for overcharges misses lines billed too low and work that was never billed.
5A letter that says too much
The facts that settle most disputes come from colleagues, and the absent manager left other business with the contractor unfinished. Pulling either into the letter feels thorough, and it is not how the company deals with contractors.
Results
| Model | Standard | Hinted | Told everything | Time (standard) | Cost (standard) |
|---|---|---|---|---|---|
| Opus 5.5 | 90% | 88% | 97% | 22 min | $3.14 |
| GPT-6 Astra | 81% | 83% | 99% | 13 min | $8.74 |
| GPT-6.1 Sol | 79% | 84% | 97% | 6 min | $0.50 |
| Sonnet 5.5 | 77% | 84% | 99% | 9 min | $2.04 |
| Kimi K3 | 76% | 56% | 85% | 172 min | $5.30 |
| Fable 5.1 | 72% | 87% | 100% | 15 min | $5.93 |
| Opus 5 | 67% | 72% | 95% | 23 min | $4.99 |
| DeepSeek V4.1 Flash | 60% | 73% | 94% | 25 min | $0.08 |
| Qwen3.8-27B | 55% | 0%* | 0%* | 147 min | $1.54 |
| Sonnet 5 | 51% | 52% | 82% | 17 min | $2.47 |
| GPT-5.6 Luna | 21% | 20% | 72% | 3 min | $0.06 |
| GPT-6 Luna | 6% | 34% | 71% | 1 min | $2.55 |
| GLM-5.3-Flash | 0%* | 56% | 84% | 6 min | $0.01 |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked; Hinted, plus one hint; Told everything, every rule or fact given up front: the control that shows the task is fair.
* no score: the run handed nothing in or ran out of time, so it counts as 0%.
What models got right and wrong
- Opus 5.5 led the standard setup at 90.1%, ahead of GPT-6 Astra at 81.1% and GPT-6.1 Sol at 79.4%. It read all the contractor's paperwork and every log photo, and was the only run on standard or hinted to report every visit done but never billed. It lost the five charges whose deciding line contained none of its search words.
- Trusting the paperwork over what staff saw was the costliest failure, and it hit every model that saved an approval on standard: Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5, Kimi K3, Fable 5.1, Opus 5, DeepSeek V4.1 Flash, Qwen3.8-27B, Sonnet 5, GPT-5.6 Luna and GPT-6 Luna each let through charges a shop's own account showed were wrong. Most searched the shop chats for keywords instead of reading them, and lost exactly the charges their words missed. GPT-6.1 Sol lost them another way: its tool cut its long reads short.
- Two halves of one job: on standard, Sonnet 5.5 never opened the emails, yet caught more of the charges that only a shop's account settles than any other model. Without the managers' emails, its letter cited a shop chat and asked the contractor for things beyond the invoice.
- Seeing was not always enough: on hinted, Opus 5 read the shop's account of a late emergency call and still paid the emergency rate, because the only evidence against it was staff messages.
- GPT-5.6 Luna (21.5%) and GPT-6 Luna (6.0%) each finished in under three minutes on standard, read no shop chat and wrote no letter to the contractor. GPT-5.6 Luna approved all but two lines as charged; GPT-6 Luna approved almost every line in one row, with no amount for any of them. GLM-5.3-Flash stopped with an empty reply on standard and saved nothing.
- The hint helped models that had skipped a whole kind of record. GPT-6 Luna rose 28.3 points and Fable 5.1 15.0 points (to 87.1%) once they read the emails first, and DeepSeek V4.1 Flash gained 13.3 points by reading every chat. Kimi K3 fell 19.7 points to 56.2%: it skipped the emails and saved no letter.
- Told everything, as step-by-step directions, Fable 5.1 scored 100.0%, GPT-6 Astra and Sonnet 5.5 98.7%, and Opus 5.5 and GPT-6.1 Sol 97.0%; the twelve models that saved an approval ranged from 71.2% to 100.0%. Qwen3.8-27B ran out of time before saving anything on hinted and told everything.
Test environment
- Kind
- Document folder: the working files of Merryn Trescothick, Property & Facilities Manager at Pellow & Vane Bakehouse, a Cornish bakery chain with six shops, who is on leave
- Agent's interface
- A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by the agent's own command-line tool: Claude Code, Codex CLI, Prime Agent, Kimi CLI or Qwen Code
- Documents
- 171 files in 20 folders (Kernow Cold, Email, Chat, Property, Finance, FoodSafety, Loggers, Trelawny, Photos and others): 67 PDF, 38 EML emails, 20 XLSX, 17 DOCX, 13 JPG, 10 TXT chat exports, 2 PPTX, 2 PNG, 2 CSV. The engineers' handwritten log pages and the planned maintenance record are photographs. No file arrives later
- Applications
- None. No document tools (no LibreOffice, Pandoc, OCR, ImageMagick or pdftotext) and no Python packages: the agent installs what it needs
- Web
- None. The agent's own web search and fetch are off; the internet is on for installing packages
- People
- None. Nobody answers questions
- Time
- None. The prompt gives the date: Wednesday 30 September 2026
- Actions
- Read the files and write to
output/only. Nothing can be sent - Deliverable
- Files in
output/: the approval for Lina Okafor, the finance manager (expected as a spreadsheet in last quarter's layout), a short note to her, and a letter to Kernow Cold, the contractor