AsapiThe AI Agent Trials
Asapi / Catalogue / Bakery wholesale pack with downstream figure checks
Environment datasheet · Small business

Bakery wholesale pack with downstream figure checks

ID house‑rules/task06 · revision 3 · tested 24–25 September and 15 October 2026 · regraded 17 November 2026

Covering for a wholesale manager on leave, the agent builds the month's wholesale pack as an Excel workbook in the house layout, with a note for the head of operations that must volunteer what's wrong in last month's pack and in other teams' figures.

Checks
23, worth 66 points
Tested
13 models, one attempt each
Best
90% (GPT‑6.1 Sol)
Median
48%
Full solves
0 of 13
Median attempt
13 min, 30 tool calls
What it tests

The month's wholesale pack, and what's wrong in other teams' figures

Catching errors

Planted faults sit in last month's figures, a customer's rebate and another team's ordering formula, and false alarms about right figures fail.

Unstated requirements

The pack follows past packs, a style guide and the owner's latest word on totals, given once in a staff chat.

How it works

53 files, an Excel workbook and teams that rely on it

The agent's world

A terminal in a fresh container with 53 files and no document or spreadsheet tools, through the model's usual coding agent.

Who relies on it

The production plan takes its trade figures from the pack, stores' flour orders follow the plan, and finance works out rebates.

Limits

One session of 180 minutes with a spend cap, and nobody to ask; the request never says check, error or verify.

The reward

23 checks worth 66 points: scripts decide 10 and two AI judges 13; a finding counts only with its figure given and right.

Results

The best two reached 77% and 90%; the rest scored 23–59%

Each dot is one attempt, and the line marks the median.

MedianFully solved
13 modelsOne attempt each48%0 of 13
ModelReward
GPT‑6.1 SolOpenAI90%
GPT‑6 AstraOpenAI77%
Opus 5.5Anthropic59%
Sonnet 5.5Anthropic59%
DeepSeek V4.1 FlashDeepSeek52%
Fable 5.1Anthropic52%
GPT‑6 LunaOpenAI48%
Opus 5Anthropic42%
GPT‑6 SolOpenAI40%
Qwen3.8‑27BAlibaba33%
Sonnet 5Anthropic33%
GLM‑5.3‑FlashZhipu23%
Kimi K3Moonshot23%

With a step‑by‑step method that gives no facts, 6 of the 13 fully solved it.

What models get wrong

They follow last month's template, trust a formula untested, and leave out methods

Unstated requirements

  1. 92% of attempts kept in the totals an account the owner had said, once in a staff chat, stays out; several quoted her rule, then read it narrowly.

  2. No note explained how stores work out their order, though the head of operations had asked notes to say how each team gets its number.

Catching errors

  1. 85% missed that stores' flour order comes out too big in five‑week months.

  2. Every attempt noticed last month's total had left out a customer, but 23% gave the gap only in pounds, not in the units the check needs, and only 46% got credit for telling the planner, which needs them too.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $1.40 at list prices for the agent model; the judges' calls are extra.

What's in it

The 53 files, the request in all three versions, the full grader with its self‑test, and per‑check results for every attempt.

Format

Packaged as a Harbor task on request; it runs in Docker today, and grading needs two judge models.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.