Brewery input‑price report in house format

58 files and a report that goes on to the bank

Role

Stand‑in for a brewery's purchasing manager, who is on leave, working from the manager's files.

Request

A colleague asks for last month's input‑price report for the managing director, saved as a Word document, from the manager's working files. Nobody is available to answer questions.

Deliverable

One Word report in the output folder, which goes from the director into the bank's review pack as it is.

What the reward depends on

Six inputs priced per pint brewed, each clash settled by the record that decides it; the unpriced lot left out and named before or beside the figures; the director's current layout over the old template; nothing meant for other readers.

Where the requirements are

In the director's reactions to past reports, her markup on a scan, chats and emails that overrule a meeting, and a whiteboard photo. An audit traced all 51 checks to the files; three graded points leave a second reasonable reading.

A folder of files, a bare container and nobody to ask

Interface

A terminal in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Prime Agent or Qwen Code in our attempts). Web search and fetch are off; the internet is on for installing packages.

Tools

None for documents: no office suite, PDF or text‑recognition tools and no Python packages. The agent installs what it needs, or uses its harness's own image viewer.

Files

58 in 17 folders: 18 emails, 13 Word files, 11 PDFs, 7 spreadsheets, 7 photos and 2 text files. One PDF is a scan with no text layer.

Evidence in images

A brew sheet survives only as a photo; delivery notes are photos, one of them settling a look‑alike double count; the director's markup on last month's report is an image‑only scan; a format rule is on a whiteboard photo.

Data problems

A returned load, a delivery booked twice beside a genuine look‑alike, a withdrawn surcharge, a price disputed between a chat and an invoice, brews made for another company, and one lot with no price yet.

People and time

Nobody answers, and there's no clock: one session, with the date set in the prompt.

Limits

180 minutes and a $25 spend cap per attempt; no attempt came near either.

51 checks, with 60% of the reward on what to leave out

Structure

51 yes/no checks in four layers, equal weight within each: the file 5% (2 checks), the figures 20% (7), the director's other requirements 15% (28), and what to leave out 60% (14).

Weights

Each thing to leave out is worth about 4.3 points, each requirement about 0.5. In the grader's self‑test, a correct report that volunteers everything but keeps to a page scores 49%; one failing every restraint check, 40%.

Guards

Checks a blank page would pass also need a substantive report, with pence figures for at least four of the six inputs. Giving both a superseded and a settled answer fails. Figures must be within 0.1p.

Who decides

Scripts decide 9 checks, 12.5% of the points: the file, length, staff and supplier names, codes, table width, input order and number format. Two LLM judges decide 42, extracting the figures a script scores and answering yes/no questions; both must agree.

Judges

Opus 4.8 and GPT‑6.1 Sol, which is also one of the models tested here. Either judge can run on a model you choose.

Rulings

The judges split on 64 of 1,512 judged results (about 4%, against 3.5% across the office and household environments), in 32 of 36 attempts; Claude settled each by written ruling for the author, 48 for Opus 4.8's reading. Rulings also overturned 8 agreed verdicts the report's words contradict.

Self‑test

The grader's ideal report scores 1 and an empty output folder 0.

Formula

Reward = the weighted share of checks passed, from 0 to 1, with no penalties.

Most figures were right; the reports around them weren't

With only the request, 9 of 12 attempts got every figure right, but the best scored 90%: what the reports said around the figures cost most. Told the rules as a colleague's method, with none of the report's figures, 4 of 12 fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version. Kimi K3 isn't on this board: its attempt with a hint never finished.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from its files, with examples from another company, and a line asking it to check every figure.

Told the rules: the request plus a colleague's ten‑step method for the job. It gives none of the report's figures.

What the models got wrong

Counted over the 12 models, one attempt each, with only the request, unless a line says otherwise.

1. The workings shown. 10 of 12, the top three among them, told the director how the figures had been checked or corrected, which she had said is for the finance team. The other 2 kept it out.

2. The old template followed. 9 of 12 gave a verdict on last month's recommendations, as the outdated template asks, though the director had since objected to it.

3. Names the bank shouldn't see. 3 of 12 named a supplier in a report bound for the bank, after a supplier had complained about exactly that; 6 of 12 added figures for the group's pubs, a separate company.

4. True but unwanted. Every attempt included at least one thing the director keeps out of this report; the 14 checks on what to leave out failed 63 times in all on our board. Told the rules, 3 times.

What an attempt costs

Length

With only the request, the median attempt took 8 minutes and 35 tool calls (21 to 54 across the 12), and about 1.0 million tokens, cached input included.

Cost

The median attempt cost $0.50 at list prices, from 4 cents to $3.82 across the 12; all 36 attempts cost $42. That's the agent model only: the judges' calls aren't included. Three GPT models with no published Bedrock price are costed at OpenAI's list price plus 10%.

What you license

With a licence

Revision 5 in all three versions: the 58 files, the grader with its judge questions and written rulings, the answer key, and the grader's report on each of the 36 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today the agent runs in a Docker container; the harness and grader are standard‑library Python, plus one package to read Word files.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• A shortcut still pays: on a test copy, a bare table with its total, one note and a recommendation scored 97%, above every attempt with only the request. Cutting four requested notes from a full solve costs less than one extra true sentence. No attempt of ours was that bare.
• 42 of the 51 checks depend on LLM judges, and one of the two judges, GPT‑6.1 Sol, is also a model under test. We haven't measured the reward under other judges.
• An audit on 9 October loosened three checks; its re‑grades aren't on our board yet. They would raise several attempts with only the request, GPT‑6 Astra's and Fable 5.1's from 90% to 95%, and make 6 of 12 attempts told the rules full solves, against 4.
• One open point stands by the author's decision: a single line on a withdrawn supplier charge counts as showing the workings, and fails. The audit found the files support either reading.
• In the version told the rules, two of the ten steps leave one narrow layout that meets both; seven attempts put the unpriced‑lot note a paragraph late, which the re‑grade would accept.
• No red‑team attempts yet.