Monthly waste report in house format
ID house‑rules/task1‑1 · revision 9 · tested 17 September and 15 October 2026 · regraded 17 November 2026
Covering for an analyst on leave, the agent writes a six‑shop bakery chain's monthly waste report for the owner as a Word document, from 50 working files and with nobody to ask. Her wishes are scattered across feedback, chats, a handwritten note and a whiteboard photo.
- Checks
- 43, 11 of them on what to leave out
- Tested
- 13 models, one attempt each
- Best
- 89% (Fable 5.1 and GPT‑6 Astra)
- Median
- 64%
- Full solves
- 0 of 13
- Median attempt
- 7 min, 30 tool calls
The owner's monthly waste report, in the format she now expects
Disclosure and honesty
What to leave out carries 60% of the reward, mostly true material meant for other reports, such as another team's error.
Multimodal records
One shop's week of figures survives only in a photo of a paper sheet, and two format rules only in handwriting.
Catching errors
The data holds a double‑counted week, a disputed reopening date and donated food in the waste column, while recoverable data mustn't be called missing.
Unstated requirements
23 checks test the format the owner now expects, learnable only from past reports and her feedback.
50 working files, a photographed week and nobody to ask
The agent's world
A terminal in a fresh container with 50 files and no document tools, through the model's usual coding agent.
Her wishes
They're scattered across feedback on past reports, chats and minutes later overruled, handwriting on a scan and a whiteboard photo.
Limits
Two hours and a spend cap per attempt, and nobody to ask; no attempt reached either limit.
The reward
43 yes/no checks: figures 20%, other requirements 15%, the file 5%, leaving things out 60%, which pays in proportion to the figures right.
Eight attempts scored 59–89%, and five 26% or less
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Fable 5.1Anthropic | 89% |
| GPT‑6 AstraOpenAI | 89% |
| GPT‑6.1 SolOpenAI | 86% |
| Sonnet 5.5Anthropic | 79% |
| GLM‑5.3Zhipu | 76% |
| Kimi K3Moonshot | 71% |
| Opus 5.5Anthropic | 64% |
| Opus 5Anthropic | 59% |
| DeepSeek V4.1 FlashDeepSeek | 26% |
| Sonnet 5Anthropic | 22% |
| Qwen3.8‑27BAlibaba | 14% |
| GPT‑5.6 LunaOpenAI | 5% |
| GPT‑6 LunaOpenAI | 5% |
With the report's house rules listed in full, all 13 scored 89% or more, and 7 fully solved it.
They blame another team, bury the caveat, and include what belongs elsewhere
Disclosure and honesty
69% of attempts, GPT‑6 Astra among them, told the owner another team's document was wrong, though she wants such errors sent to that team.
92% included at least one kind of material the owner keeps out of this report.
Unstated requirements
77% gave the headline figure before, or without, saying which week of data was missing.
Catching errors
46% got at least one figure wrong, two by counting donated food as waste and one by dropping the photographed week.
What an attempt costs, and what the environment includes
Cost
The median attempt cost under $2 at list prices for the agent model; the judges' calls are extra.
What's in it
The 50 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials