Bakery wholesale pack with downstream figure checks
ID house‑rules/task06 · revision 3 · tested 24–25 September and 15 October 2026 · regraded 17 November 2026
Covering for a wholesale manager on leave, the agent builds the month's wholesale pack as an Excel workbook in the house layout, with a note for the head of operations that must volunteer what's wrong in last month's pack and in other teams' figures.
- Checks
- 23, worth 66 points
- Tested
- 13 models, one attempt each
- Best
- 90% (GPT‑6.1 Sol)
- Median
- 48%
- Full solves
- 0 of 13
- Median attempt
- 13 min, 30 tool calls
The month's wholesale pack, and what's wrong in other teams' figures
Catching errors
Planted faults sit in last month's figures, a customer's rebate and another team's ordering formula, and false alarms about right figures fail.
Unstated requirements
The pack follows past packs, a style guide and the owner's latest word on totals, given once in a staff chat.
53 files, an Excel workbook and teams that rely on it
The agent's world
A terminal in a fresh container with 53 files and no document or spreadsheet tools, through the model's usual coding agent.
Who relies on it
The production plan takes its trade figures from the pack, stores' flour orders follow the plan, and finance works out rebates.
Limits
One session of 180 minutes with a spend cap, and nobody to ask; the request never says check, error or verify.
The reward
23 checks worth 66 points: scripts decide 10 and two AI judges 13; a finding counts only with its figure given and right.
The best two reached 77% and 90%; the rest scored 23–59%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6.1 SolOpenAI | 90% |
| GPT‑6 AstraOpenAI | 77% |
| Opus 5.5Anthropic | 59% |
| Sonnet 5.5Anthropic | 59% |
| DeepSeek V4.1 FlashDeepSeek | 52% |
| Fable 5.1Anthropic | 52% |
| GPT‑6 LunaOpenAI | 48% |
| Opus 5Anthropic | 42% |
| GPT‑6 SolOpenAI | 40% |
| Qwen3.8‑27BAlibaba | 33% |
| Sonnet 5Anthropic | 33% |
| GLM‑5.3‑FlashZhipu | 23% |
| Kimi K3Moonshot | 23% |
With a step‑by‑step method that gives no facts, 6 of the 13 fully solved it.
They follow last month's template, trust a formula untested, and leave out methods
Unstated requirements
92% of attempts kept in the totals an account the owner had said, once in a staff chat, stays out; several quoted her rule, then read it narrowly.
No note explained how stores work out their order, though the head of operations had asked notes to say how each team gets its number.
Catching errors
85% missed that stores' flour order comes out too big in five‑week months.
Every attempt noticed last month's total had left out a customer, but 23% gave the gap only in pounds, not in the units the check needs, and only 46% got credit for telling the planner, which needs them too.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $1.40 at list prices for the agent model; the judges' calls are extra.
What's in it
The 53 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials