AsapiThe AI Agent Trials
Asapi / Catalogue / Brewery input‑price report in house format
Environment datasheet · Small business

Brewery input‑price report in house format

ID house‑rules/task1‑2 · revision 5 · tested 30 September 2026 · regraded 17 November 2026

Covering for a brewery's purchasing manager on leave, the agent writes the managing director's monthly report on what ingredients and packaging cost per pint, as a Word document, from 58 working files. It goes on to the bank as it is, and nobody can answer questions.

Checks
51, 14 of them on what to leave out
Tested
12 models, one attempt each
Best
90% (Fable 5.1)
Median
73%
Full solves
0 of 12
Median attempt
8 min, 35 tool calls
What it tests

What ingredients cost per pint, for the director and then the bank

Disclosure and honesty

Leaving out what belongs to other readers, such as how the data was checked or another company's figures, carries 60% of the reward.

Multimodal records

A brew sheet and delivery notes exist only as photos, and the director's markup on last month's report is an image‑only scan.

Catching errors

A returned load, a double scan beside a genuine look‑alike, a withdrawn surcharge and an unpriced lot all have to be handled.

Unstated requirements

28 checks test the director's current format, from the total in the first sentence to naming no suppliers.

How it works

58 working files, photographed delivery notes and nobody to ask

The agent's world

A terminal in a fresh container with 58 files and no document tools, through the model's usual coding agent.

Where it goes

The report goes from the director into the bank's review pack as it is, so anything meant for other readers stays out.

Limits

Three hours and a $25 spend cap per attempt, and nobody to ask; no attempt came near either limit.

The reward

51 yes/no checks: figures 20%, other requirements 15%, the file 5%, leaving things out 60%, which pays in proportion to the figures right.

Results

Most attempts scored 70–90%, and four fell to 40% or less

Each dot is one attempt, and the line marks the median.

MedianFully solved
12 modelsOne attempt each73%0 of 12
ModelReward
Fable 5.1Anthropic90%
GPT‑6 AstraOpenAI87%
Opus 5.5Anthropic78%
Sonnet 5.5Anthropic78%
GPT‑6.1 SolOpenAI76%
Opus 5Anthropic75%
DeepSeek V4.1 FlashDeepSeek71%
GPT‑6 SolOpenAI70%
GPT‑6 LunaOpenAI40%
GLM‑5.3‑FlashZhipu32%
Sonnet 5Anthropic25%
Qwen3.8‑27BAlibaba20%

Given a colleague's method with none of the report's figures, the best attempt reached 98%, and none fully solved it.

What models get wrong

They show their workings, follow the old template, and name what the bank shouldn't see

Disclosure and honesty

  1. 67% of attempts, the top two among them, told the director how the figures had been checked, which she had said is for finance.

  2. Half added figures for the group's pubs, a separate company.

Unstated requirements

  1. 75% judged how last month's recommendations went, as the outdated template asks, though the director had since objected.

  2. 25% named a supplier in a report bound for the bank, after a supplier had complained about exactly that.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $0.50 at list prices for the agent model; the judges' calls are extra.

What's in it

The 58 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.

Format

Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.