Bakery wholesale pack with downstream figure checks

The request asks for a pack and a note; most of the reward is for what it volunteers

Role

Stand‑in for the wholesale manager of a bakery chain that sells to eight trade accounts. Nine of our environments are set at this chain, each a different job.

Request

Do the absent manager's monthly wholesale pack, saved as an Excel workbook, with a note for the head of operations. Nobody can answer questions. The request never says check, error or verify.

Deliverable

An Excel workbook, recalculated in LibreOffice before it's read, and a note. The judges read every file the agent saved.

What the reward depends on

Totals that follow the owner's latest word on what they leave out; faults found in last month's pack and in other teams' figures, with their size and who must act; how each team works out its figure; and no false alarms about figures that are right.

Where the requirements are

In past packs, a how‑to note, a style guide, rebate schedules, production plans, flour orders, emails, minutes and a staff chat. The owner's change to the totals appears once, in the chat; last month's pack never caught up.

53 files of nested tables, no images, and nobody to ask

Interface

A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts).

Tools

None for documents or spreadsheets: no office suite, PDF text tool or OCR, and no Python packages. The agent installs what it needs; the internet is on for that, and the harness's own web search and fetch are off.

Files

53 in 8 folders, 0.5 MB: 31 PDFs, most of them reports with tables several levels deep, 13 emails, 3 Word files, 3 spreadsheets, a staff chat, the order system's export and a slide deck. No images.

Who relies on it

The production plan takes its trade figures from the pack, and stores' flour order follows the plan. Finance works out quarterly rebates from its own ledger.

Traps

Last month's working sheet left a customer out of its total; one team's method is right in most months but not all; a stop‑gap outlived its reason; and some odd‑looking figures are right.

People

None. The request says nobody can be asked. A version where the agent can message colleagues exists in the harness, but none of our attempts used it.

Time

A single session; the request gives the date. Each attempt had 180 minutes and a spend cap: $25, or $10 and 150 turns for the two models tested on 15 October.

What the grader reads

Only the output folder. Each workbook is recalculated in LibreOffice and shown to the judges row by row, so a formula saved without its value is read as its result.

23 checks, 22 of them yes/no questions to two judges

Structure

23 checks worth 66 points, weighted 1 to 5: the pack's totals and three accounts' rows; four faults, three of them with their size and who must act; how three teams work out their figures; three no‑false‑alarm checks; and one script check that the pack is a workbook.

Dependencies

Six checks count only when the finding they rest on was made: who must act on a fault, and the no‑false‑alarm checks. An attempt that finds nothing can't collect them.

No hedging

A note that says a correct rebate might be wrong, or asks finance to check it, fails that no‑false‑alarm check.

Who decides

One script check; two LLM judges decide 22 as yes/no questions with every saved file: Opus 4.8 and GPT‑5.6 Luna, or Sonnet 5.5 and GPT‑5.6 Luna for the two tested on 15 October. An unreadable reply is never a pass. Either judge can run on a model you choose.

Judge agreement

The judges agreed on 870 of 874 answers across the three versions; the 4 splits were settled by written ruling, by Claude for the environment's author.

Formula

Reward = points earned / 66, from 0 to 1, with no gate or penalty.

One attempt followed the owner's rule; told the method, 7 of 13 fully solved it

With only the request, the 13 attempts, one per model, scored a median of 56% and at best 91% (GPT‑6.1 Sol); none fully solved it, with or without a hint. Told the rules, a method with no facts, 7 of 13 did; DeepSeek V4.1 Flash saved nothing.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from its files, with examples from an unrelated company, and a line to check every figure.

Told the rules: the request plus a colleague's seven step‑by‑step directions, never an answer.

What the models got wrong

Counted over the 13 models, one attempt each, with only the request, unless a line says otherwise.

1. Last month's template over the owner's word. 12 of 13 attempts kept in the totals an account the owner had said, once in a staff chat, stays out of every total; only GPT‑6.1 Sol left it out. Several quoted her rule, then read it narrowly.

2. A method trusted, not tested. 11 of 13 attempts missed that stores' flour order comes out too big in five‑week months; GPT‑6 Astra and GPT‑6.1 Sol tested it across months and found it. Told to check a month of a different length, 10 of 13 found it.

3. Figures without methods. No attempt explained how stores work out their order, and 1 of 13 how the planner does, though the head of operations had asked notes to say how each team gets its number. Told the rules, 12 and 9 did.

4. Found but not passed on. All 13 attempts found that last month's total had left out a customer, but only 7 said the planner, who had already used the short figure, needed it corrected.

5. The hint cut both ways. With a hint, Fable 5.1 and Kimi K3 followed the owner's rule, while GPT‑6.1 Sol argued itself out of it and fell from 91% to 79%; no attempt fully solved it.

What an attempt costs

Length

With only the request, the median attempt took 13 minutes and 30 tool calls (14 to 112 across the 13), and about 1.7 million tokens, cached input included.

Cost

The median attempt cost $1.39 at list prices, from $0.06 to $4.89 across the 13; all 39 attempts cost $79. That's the agent model only: the judges' calls aren't included.

What you license, and what's public

Format

Packaged as a Harbor task on request. It isn't one of our three downloadable samples.

With a licence

The 53 files, the request in all three versions, the grader with its judge prompts and self‑test, the answer key, and the grader's report on each of the 39 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• 22 of the 23 checks depend on LLM judges, and Sonnet 5.5 judged its own attempts with GPT‑5.6 Luna; the two agreed on every answer for the two models tested on 15 October.
• The three total checks turn on one choice, whether the excluded account is in, so that choice is worth 9 of the 66 points.
• One check accepts last month's shortfall only in units: five attempts that gave the right amount in money failed it, 3 points each. The author kept it by design, since the plan works in units.
• The told‑the‑rules steps point close to each fault. That version is a control that the job can be done, not the main result.
• Doing nothing scores 0 in the grader's self‑test, where a minimal pack with every judge answer mocked as right scores the maximum.
• No red‑team attempts yet.