Monthly waste report in house format

50 files, nobody to ask, and an owner's scattered wishes

Role

Stand‑in for a bakery chain's operations analyst, who is on leave, working from the analyst's files.

Request

A manager asks for last month's waste report for the owner, saved as a Word document, from the analyst's working files. Nobody is available to answer questions.

Deliverable

One Word report in the output folder.

What the reward depends on

Each shop's waste as a share of what it baked, through the data's glitches; saying what's missing before the figures it affects; the owner's current layout over the old template; and leaving out true material that belongs in other reports.

Where the requirements are

Nowhere in one place: in the owner's reactions to past reports, in chats and minutes later overruled, in handwriting on a scan and on a whiteboard photo. An audit traced each of the 35 requirements to the files; only an exact word limit and one placement rule aren't settled there.

A folder of files, a bare container and nobody to ask

Interface

A terminal in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts). Web search and fetch are off; the internet is on for installing packages.

Tools

None for documents: no office suite, PDF or text‑recognition tools and no Python packages. The agent installs what it needs, or uses its harness's own image viewer.

Files

50 in 16 folders: 15 Word files, 9 spreadsheets, 8 PDFs, 8 emails, 6 photos, 2 slide decks and 2 text files. One PDF is a scan with no text layer.

Evidence in images

One shop's week of figures survives only in a photo of a paper sheet; two of the owner's format rules appear only in handwriting on a scan and on a whiteboard photo.

Data glitches

A week counted twice when the tills changed, a reopening date two files disagree on, a week never counted, and donated food sitting in the waste column.

People and time

Nobody answers, and there's no clock: one session, with the date set in the prompt.

Limits

120 minutes and a spend cap per attempt; no attempt reached either.

43 checks, with 60% of the reward on what to leave out

Structure

43 yes/no checks in four layers, equal weight within each: the file 5% (2 checks), the figures 20% (7), the owner's other requirements 15% (23), and what to leave out 60% (11).

Why restraint weighs most

A report with every figure and requirement right that breaks every rule on what to leave out scores 40%. In the grader's self‑test, a correct report that volunteers most of what belongs elsewhere scores 62%.

Guards

Checks a blank page would pass also need a substantive report, with figures for at least four shops. Giving both a superseded and a settled answer fails. Figures must be within 0.1 points.

Who decides

Scripts decide 8 checks, 14% of the points: the file, length, names, codes, table width, shop order and number format. Two LLM judges decide 35, extracting the figures a script scores and answering yes/no questions; both must agree.

Judges

Opus 4.8 and GPT‑5.6 Sol for nine models' attempts; Sonnet 5.5 and GPT‑5.6 Luna for the four models added on 15 October. Either judge can run on a model you choose.

Rulings

The judges split on 22 of 1,365 judged results (about 1.6%, against 3.5% across the office and household environments), in 18 of 39 attempts; each split was settled by written ruling. Rulings also overturned 3 passes both judges gave that the report's own figures contradict.

Self‑test

The grader's ideal report scores 1 and an empty output folder 0; last month's report copied forward scores 4%.

Formula

Reward = the weighted share of checks passed, from 0 to 1, with no penalties.

The figures were right more often than the reports around them

With only the request, 7 of 13 attempts got every figure right, yet the best scored 94%: the losses were in what each report said around its figures. Told the rules, all 13 scored 94% or more, and 8 fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version; two that ended on provider errors were run again.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from patterns across its files, with examples from another company.

Told the rules: here, the request plus every requirement as a plain list, including facts from the files and one corrected figure the grader checks.

What the models got wrong

Counted over the 13 models, one attempt each, with only the request, unless a line says otherwise.

1. Another team's error in the owner's report. 9 of 13, the top four among them, told the owner another team's document was wrong to explain their own figure, though she had asked for such mistakes to go to that team. Of the 4 that didn't, one passed it on.

2. The total before the caveat. 10 of 13 gave the headline figure before, or without, saying which week of data was missing, though the owner had once passed on a total before reaching the note at the bottom.

3. Rules noted, then broken. 2 of 13, both in the top four, wrote in their own notes that other teams' errors go to that team, then named one in the report.

4. True but unwanted. 12 of 13 included at least one kind of material the owner keeps out of this report; the 11 checks on what to leave out failed 43 times in all on our board. Told the rules, 4 times.

5. Wrong figures. 6 of 13 got at least one figure wrong: two counted donated food as waste, two built the report from the analyst's unfinished draft, and one judged the photographed week a partial count and dropped it.

What an attempt costs

Length

With only the request, the median attempt took 7 minutes and 30 tool calls (13 to 102 across the 13), and about 1.4 million tokens, cached input included.

Cost

The median attempt cost $1.65 at list prices, from $0.05 to $4.32 across the 13; all 39 attempts cost $64. That's the agent model only: the judges' calls aren't included. GPT‑6 Luna has no published price, so its cost is an upper estimate.

What you license

With a licence

Revision 9 in all three versions: the 50 files, the grader with its judge questions and written rulings, the answer key, and the grader's report on each of the 39 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today the agent runs in a Docker container; the harness and grader are standard‑library Python, plus one package to read Word files.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• A shortcut still pays. On test copies, a title, the period and one table of correct figures scored 84%, above 9 of 13 attempts with only the request; one real report cut to its two tables rose from 65% to 89%. None of ours was that bare.
• 35 of the 43 checks depend on LLM judges; one judge disagrees with each of the 22 rulings. Sonnet 5.5 judged its own three attempts, with GPT‑5.6 Luna. We haven't measured the reward under other judges.
• The first nine models' splits were ruled by Claude for the author, and some by the author; the record of the 9 rulings on the four models added on 15 October doesn't say who made them.
• The 450‑word limit is a cut‑off the author chose between a past report the owner accepted and one she gave up on; the files give no number.
• An audit on 9 October corrected three checks; its re‑grades aren't on our board yet. With only the request, they would take Fable 5.1 from 83% to 94%, level with the best, DeepSeek V4.1 Flash to 71%, GPT‑5.6 Luna to 42% and Sonnet 5 to 68%.
• No red‑team attempts yet.