Packaging proof sign‑off for an absent owner

The request asks for a pick and a signature; the files decide whether to sign

Role

Stand‑in for the marketing manager of a bakery chain, who is on leave, while the owner is out all day. Nine of our environments are set at this chain, each a different job.

Request

Pick the design for the new party‑cake box that the owner would pick, and fill in the printer's forms in her name. Nobody can answer questions. The request never says to check, approve or reject; the files show the forms are due back that day.

Deliverable

The printer's two approval forms, filled in as Word files: one for the new box, one for an unchanged reprint of an older box. Any other file saved is read as a note alongside them.

What the reward depends on

The right decision on each form, judged on its own; the reasons, written where the printer will see them, not in the chat reply; the owner's latest choice of design under the printer's current labels; and the order details, signature and date.

Where the requirements are

In 181 files: three mailboxes, chats, drawings and spreadsheets. The deciding sizes are in box proofs and a maker's product sheet, images inside PDFs attached to emails. The 9 October audit traced every graded answer to the files but one: how much room is enough.

181 files, three mailboxes, and the deciding sizes only in images

Interface

A terminal in a fresh Docker container (Debian, Python 3.13 with pip, Node 22), driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts).

Tools

None for documents: no office suite, PDF text tool, OCR or image tools, and no Python packages. The agent installs what it needs; the internet is on for that, and the harness's own web search and fetch are off.

Files

181 in 21 folders, 18.2 MB: 3 mailbox exports holding 850 emails, 34 more emails, 79 PDFs, 23 Word files, 23 spreadsheets, 8 text files (7 chat exports), 7 photos, 2 slide decks and 2 CSV files.

Images

The box proofs and the maker's product sheet are images inside attached PDFs, and the kitchen's fit test is a photographed handwritten sheet. None has text to search, so each must be looked at.

Traps

A part on one cake was swapped after the only fit test, and the seller's stated size differs from the maker's sheet; the printer relettered the designs between rounds; a later chat changed the delivery split; the deposit is paid and the press date booked.

People

None. The request says nobody can be asked; the owner and the manager are away.

Time

A single session; the forms are due that day. Each attempt had 180 minutes and a spend cap.

What the grader reads

Only the output folder. Forms are told apart by job number, not file name; a script reads each form's ticks and fields, and the judges see the forms' text with every other file saved.

15 checks, with the box decision and its reasons worth most

Structure

15 checks worth 49 points. Rejecting the box is worth 10, and nothing in the submission approving it 5; six reasons, 21 in all; approving the unchanged reprint 5; the design 3; the form, three order details and the signature 1 each.

No gate

Approving the box loses at least 20 points, so an approving attempt scores at most 29 of 49 (59%). Two reasons count only on a rejected box, so the right words with the wrong decision earn nothing there.

Separate decisions

Each form is scored on its own decision, worth 10 and 5, so one blanket call can't pass both; a form with two boxes ticked fails.

Who decides

Scripts decide 8 checks: the two decisions, the design, the form, the order details and the signature. Two LLM judges answer 7 yes/no questions with every saved file: Opus 4.8 and GPT‑5.6 Luna, or Sonnet 5.5 and GPT‑5.6 Luna for the two tested on 15 October.

Judge agreement

The judges split on 12 of 273 answers, in 9 attempts; each split was settled by written ruling, 11 of them accepted by the environment's author and the 12th with no ruler recorded. Either judge can run on a model you choose.

Formula

Reward = points earned / 49, from 0 to 1.

One model refused the box with only the request; taught the steps, all 13 did

With only the request, 12 of 13 attempts, one per model, approved the box and scored 37% or less; GPT‑6 Astra refused it and scored 94%. With a hint, 9 refused it and 4 fully solved it; told the rules, all 13 refused it and 12 fully solved it.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus a general paragraph on reading a company's unwritten expectations from its files, with examples from an unrelated company, and lines to verify every figure and to leave proper room where one thing fits inside another.

Told the rules: the request plus ten step‑by‑step directions, as a colleague would teach the job, with a minimum room to leave and no answers.

What the models got wrong

Counted over the 13 models, one attempt each, with only the request, unless a line says otherwise.

1. Signed under pressure. 12 of 13 attempts approved the new box, though the files show its tallest cake would almost touch the lid. All 13 filled in the order details and signature correctly.

2. The seller's word over the maker's sheet. 10 of 13 attempts never said the swapped part is taller than the seller claimed; 8 of them never opened the maker's product sheet, an image inside an attached PDF.

3. Warnings left in the chat. Opus 5.5 and Sonnet 5.5 saw the risk and approved to keep the print date, writing their warnings only in their chat replies; the grader, like the printer, reads only the forms and saved files.

4. No trial fit asked for. 12 of 13 attempts never asked for the cake with its new part to be measured or tried in the box before printing; only GPT‑6 Astra did.

5. The one refusal asked too little. GPT‑6 Astra asked only to restore the box's small original margin, though a box with a little room had marked a cake the year before. No attempt asked for enough room with only the request.

What an attempt costs

Length

With only the request, the median attempt took 7 minutes and 32 tool calls (14 to 62 across the 13), and about 1.7 million tokens, cached input included.

Cost

The median attempt cost $0.84 at list prices, from $0.02 to $4.80 across the 13; all 39 attempts cost $66. That's the agent model only: the judges' calls aren't included.

What you license, and what's public

Format

Packaged as a Harbor task on request. It isn't one of our three downloadable samples.

With a licence

The 181 files, the request in all three versions, the grader with its judge prompts and self‑test, the answer key, and the grader's report on each of the 39 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• 7 of the 15 checks depend on LLM judges, and Sonnet 5.5 is both a judge and a tested model; 12 splits were settled by written ruling.
• A bare rejection with no reason scores 57%, more than Fable 5.1's reasoned approval (37%); rejecting both forms loses the reprint's 5 points. The audit didn't count it as a fault, since the decision is the job's main answer.
• Doing almost nothing scores 12%: the blank forms sent back unfilled meet two checks, since nothing in them approves the box. No attempt did this.
• How much room is enough is the author's cut‑off: the files show a box with a little room still marked a cake, but give no figure. Only the told‑the‑rules version states one.
• A pair of forms filled from the key scores 49 of 49 in the grader's self‑test, with the judges' answers mocked; the self‑test has no empty‑folder case.
• No red‑team attempts yet.