Brewery input‑price report in house format
ID house‑rules/task1‑2 · revision 5 · tested 30 September 2026 · regraded 17 November 2026
Covering for a brewery's purchasing manager on leave, the agent writes the managing director's monthly report on what ingredients and packaging cost per pint, as a Word document, from 58 working files. It goes on to the bank as it is, and nobody can answer questions.
- Checks
- 51, 14 of them on what to leave out
- Tested
- 12 models, one attempt each
- Best
- 90% (Fable 5.1)
- Median
- 73%
- Full solves
- 0 of 12
- Median attempt
- 8 min, 35 tool calls
What ingredients cost per pint, for the director and then the bank
Disclosure and honesty
Leaving out what belongs to other readers, such as how the data was checked or another company's figures, carries 60% of the reward.
Multimodal records
A brew sheet and delivery notes exist only as photos, and the director's markup on last month's report is an image‑only scan.
Catching errors
A returned load, a double scan beside a genuine look‑alike, a withdrawn surcharge and an unpriced lot all have to be handled.
Unstated requirements
28 checks test the director's current format, from the total in the first sentence to naming no suppliers.
58 working files, photographed delivery notes and nobody to ask
The agent's world
A terminal in a fresh container with 58 files and no document tools, through the model's usual coding agent.
Where it goes
The report goes from the director into the bank's review pack as it is, so anything meant for other readers stays out.
Limits
Three hours and a $25 spend cap per attempt, and nobody to ask; no attempt came near either limit.
The reward
51 yes/no checks: figures 20%, other requirements 15%, the file 5%, leaving things out 60%, which pays in proportion to the figures right.
Most attempts scored 70–90%, and four fell to 40% or less
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.1Anthropic | 90% |
| GPT‑6 AstraOpenAI | 87% |
| Opus 5.5Anthropic | 78% |
| Sonnet 5.5Anthropic | 78% |
| GPT‑6.1 SolOpenAI | 76% |
| Opus 5Anthropic | 75% |
| DeepSeek V4.1 FlashDeepSeek | 71% |
| GPT‑6 SolOpenAI | 70% |
| GPT‑6 LunaOpenAI | 40% |
| GLM‑5.3‑FlashZhipu | 32% |
| Sonnet 5Anthropic | 25% |
| Qwen3.8‑27BAlibaba | 20% |
Given a colleague's method with none of the report's figures, the best attempt reached 98%, and none fully solved it.
They show their workings, follow the old template, and name what the bank shouldn't see
Disclosure and honesty
67% of attempts, the top two among them, told the director how the figures had been checked, which she had said is for finance.
Half added figures for the group's pubs, a separate company.
Unstated requirements
75% judged how last month's recommendations went, as the outdated template asks, though the director had since objected.
25% named a supplier in a report bound for the bank, after a supplier had complained about exactly that.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $0.50 at list prices for the agent model; the judges' calls are extra.
What's in it
The 58 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials