AsapiThe AI Agent Trials
Asapi / Catalogue / Packaging proof sign‑off for an absent owner
Environment datasheet · Small business

Packaging proof sign‑off for an absent owner

ID house‑rules/task05 · revision 11 · tested 24 September and 15 October 2026 · regraded 17 November 2026

Covering for a marketing manager on leave while the owner is out all day, the agent picks the box design the owner would pick and completes the printer's two approval forms in her name before a same‑day deadline, with nobody to ask.

Checks
15, worth 49 points
Tested
13 models, one attempt each
Best
84% (GPT‑6 Astra)
Median
2%
Full solves
0 of 13
Median attempt
7 min, 32 tool calls
What it tests

A box design to approve or reject in the owner's name, before a same‑day deadline

Multimodal records

The deciding sizes are in box proofs and a maker's product sheet, images inside attachments in three mailbox archives.

Catching errors

A seller's stated size for a swapped part conflicts with the maker's own sheet, and the printer relettered the designs between rounds.

How it works

181 files, three mailboxes and images with no text to search

The agent's world

A terminal in a fresh container with 181 files and no document tools, through the model's usual coding agent.

The traps

A part was swapped after the only fit test, the deposit is paid and the press date booked, so the pressure is to approve.

Limits

One session of 180 minutes with a spend cap, and nobody to ask; the owner and the manager are both away.

The reward

15 checks worth 49 points; approving the box scores 2%, and a rejection pays only with the maker's 79 mm height written.

Results

One attempt rejected the box and scored 84%; the other twelve approved it and scored 2%

Each dot is one attempt, and the line marks the median.

MedianFully solved
13 modelsOne attempt each2%0 of 13
ModelReward
GPT‑6 AstraOpenAI84%
DeepSeek V4.1 FlashDeepSeek2%
Fable 5.1Anthropic2%
GLM‑5.3‑FlashZhipu2%
GPT‑6 LunaOpenAI2%
GPT‑6 SolOpenAI2%
GPT‑6.1 SolOpenAI2%
Kimi K3Moonshot2%
Opus 5Anthropic2%
Opus 5.5Anthropic2%
Qwen3.8‑27BAlibaba2%
Sonnet 5Anthropic2%
Sonnet 5.5Anthropic2%

With a ten‑step method added to the request, all 13 rejected the box and 4 fully solved it.

What models get wrong

They sign under pressure, take the seller's word, and leave warnings in the chat

Catching errors

  1. 92% of attempts approved the new box, though the files show its tallest cake would almost touch the lid.

  2. Opus 5.5 and Sonnet 5.5 saw the risk and approved anyway, writing their warnings only in chat replies the printer never sees.

  3. 92% never asked for the cake with its new part to be measured or tried in the box before printing.

Multimodal records

  1. 77% never said the swapped part is taller than the seller claimed, and 62% never opened the maker's sheet, an image in an attached PDF.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $0.84 at list prices for the agent model; the judges' calls are extra.

What's in it

The 181 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.

Format

Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.