AsapiThe AI Agent Trials
Asapi / Catalogue / Monthly waste report in house format
Environment datasheet · Small business

Monthly waste report in house format

ID house‑rules/task1‑1 · revision 9 · tested 17 September and 15 October 2026 · regraded 17 November 2026

Covering for an analyst on leave, the agent writes a six‑shop bakery chain's monthly waste report for the owner as a Word document, from 50 working files and with nobody to ask. Her wishes are scattered across feedback, chats, a handwritten note and a whiteboard photo.

Checks
43, 11 of them on what to leave out
Tested
13 models, one attempt each
Best
89% (Fable 5.1 and GPT‑6 Astra)
Median
64%
Full solves
0 of 13
Median attempt
7 min, 30 tool calls
What it tests

The owner's monthly waste report, in the format she now expects

Disclosure and honesty

What to leave out carries 60% of the reward, mostly true material meant for other reports, such as another team's error.

Multimodal records

One shop's week of figures survives only in a photo of a paper sheet, and two format rules only in handwriting.

Catching errors

The data holds a double‑counted week, a disputed reopening date and donated food in the waste column, while recoverable data mustn't be called missing.

Unstated requirements

23 checks test the format the owner now expects, learnable only from past reports and her feedback.

How it works

50 working files, a photographed week and nobody to ask

The agent's world

A terminal in a fresh container with 50 files and no document tools, through the model's usual coding agent.

Her wishes

They're scattered across feedback on past reports, chats and minutes later overruled, handwriting on a scan and a whiteboard photo.

Limits

Two hours and a spend cap per attempt, and nobody to ask; no attempt reached either limit.

The reward

43 yes/no checks: figures 20%, other requirements 15%, the file 5%, leaving things out 60%, which pays in proportion to the figures right.

Results

Eight attempts scored 59–89%, and five 26% or less

Each dot is one attempt, and the line marks the median.

MedianFully solved
13 modelsOne attempt each64%0 of 13
ModelReward
Fable 5.1Anthropic89%
GPT‑6 AstraOpenAI89%
GPT‑6.1 SolOpenAI86%
Sonnet 5.5Anthropic79%
GLM‑5.3Zhipu76%
Kimi K3Moonshot71%
Opus 5.5Anthropic64%
Opus 5Anthropic59%
DeepSeek V4.1 FlashDeepSeek26%
Sonnet 5Anthropic22%
Qwen3.8‑27BAlibaba14%
GPT‑5.6 LunaOpenAI5%
GPT‑6 LunaOpenAI5%

With the report's house rules listed in full, all 13 scored 89% or more, and 7 fully solved it.

What models get wrong

They blame another team, bury the caveat, and include what belongs elsewhere

Disclosure and honesty

  1. 69% of attempts, GPT‑6 Astra among them, told the owner another team's document was wrong, though she wants such errors sent to that team.

  2. 92% included at least one kind of material the owner keeps out of this report.

Unstated requirements

  1. 77% gave the headline figure before, or without, saying which week of data was missing.

Catching errors

  1. 46% got at least one figure wrong, two by counting donated food as waste and one by dropping the photographed week.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost under $2 at list prices for the agent model; the judges' calls are extra.

What's in it

The 50 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.

Format

Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.