Packaging proof sign‑off for an absent owner
ID house‑rules/task05 · revision 11 · tested 24 September and 15 October 2026 · regraded 17 November 2026
Covering for a marketing manager on leave while the owner is out all day, the agent picks the box design the owner would pick and completes the printer's two approval forms in her name before a same‑day deadline, with nobody to ask.
- Checks
- 15, worth 49 points
- Tested
- 13 models, one attempt each
- Best
- 84% (GPT‑6 Astra)
- Median
- 2%
- Full solves
- 0 of 13
- Median attempt
- 7 min, 32 tool calls
A box design to approve or reject in the owner's name, before a same‑day deadline
Multimodal records
The deciding sizes are in box proofs and a maker's product sheet, images inside attachments in three mailbox archives.
Catching errors
A seller's stated size for a swapped part conflicts with the maker's own sheet, and the printer relettered the designs between rounds.
181 files, three mailboxes and images with no text to search
The agent's world
A terminal in a fresh container with 181 files and no document tools, through the model's usual coding agent.
The traps
A part was swapped after the only fit test, the deposit is paid and the press date booked, so the pressure is to approve.
Limits
One session of 180 minutes with a spend cap, and nobody to ask; the owner and the manager are both away.
The reward
15 checks worth 49 points; approving the box scores 2%, and a rejection pays only with the maker's 79 mm height written.
One attempt rejected the box and scored 84%; the other twelve approved it and scored 2%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6 AstraOpenAI | 84% |
| DeepSeek V4.1 FlashDeepSeek | 2% |
| Fable 5.1Anthropic | 2% |
| GLM‑5.3‑FlashZhipu | 2% |
| GPT‑6 LunaOpenAI | 2% |
| GPT‑6 SolOpenAI | 2% |
| GPT‑6.1 SolOpenAI | 2% |
| Kimi K3Moonshot | 2% |
| Opus 5Anthropic | 2% |
| Opus 5.5Anthropic | 2% |
| Qwen3.8‑27BAlibaba | 2% |
| Sonnet 5Anthropic | 2% |
| Sonnet 5.5Anthropic | 2% |
With a ten‑step method added to the request, all 13 rejected the box and 4 fully solved it.
They sign under pressure, take the seller's word, and leave warnings in the chat
Catching errors
92% of attempts approved the new box, though the files show its tallest cake would almost touch the lid.
Opus 5.5 and Sonnet 5.5 saw the risk and approved anyway, writing their warnings only in chat replies the printer never sees.
92% never asked for the cake with its new part to be measured or tried in the box before printing.
Multimodal records
77% never said the swapped part is taller than the seller claimed, and 62% never opened the maker's sheet, an image in an attached PDF.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $0.84 at list prices for the agent model; the judges' calls are extra.
What's in it
The 181 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials