AsapiThe AI Agent Trials
Asapi / Catalogue / 12‑month equipment plan with simulated shop manager
Environment datasheet · Small business

12‑month equipment plan with simulated shop manager

ID house‑rules/task07 · revision 4 · tested 28–29 September and 15 October 2026 · regraded 17 November 2026

A bakery shop's manager messages that his ice cream cabinet has failed and asks which new model to buy. With one catalogue and messages to him, the agent plans what to order over the next 12 months, scored by the shop's profit.

Checks
Cost model, the plan's value, 0 at the manager's own plan, times the share of 7 key facts it quotes
Tested
13 models, one attempt each
Best
57% (GPT‑6 Astra)
Median
0%
Full solves
0 of 13
Median attempt
3 min, 8 tool calls
What it tests

Which cabinet to buy, or whether to buy one at all

Clarifying with people

The manager answers from a short, partly wrong memory and reads out his records only when asked, within ten messages the agent isn't told about.

Planning ahead

A profit model prices the plan over 12 months, from the manager's own plan at zero to the best of 51,121 candidates.

How it works

One catalogue, one manager to message, and a profit model

The agent's world

A terminal in a fresh container with one catalogue of 11 cabinets and freezers, and one command to message the manager.

The manager

An AI model voices him from a 19‑line memory and 31 records; code decides what he looks up and checks every number he gives.

Hard rules

Breaking a shop rule, such as a cabinet too big for its bay, scores 0, as does never getting the engineer's visit notes from him.

The reward

The plan's 12‑month profit, scaled 0 to 1, times the share of 7 key facts its quotes of the manager carry word for word.

Results

The best attempt scored 57%; most bought what the manager asked for and scored 0

Each dot is one attempt, and the line marks the median.

MedianFully solved
13 modelsOne attempt each0%0 of 13
ModelReward
GPT‑6 AstraOpenAI57%
Opus 5.5Anthropic7%
GPT‑6.1 SolOpenAI6%
Opus 5Anthropic6%
DeepSeek V4.1 FlashDeepSeek0%
Fable 5.1Anthropic0%
GLM‑5.3‑FlashZhipu0%
GPT‑6 LunaOpenAI0%
GPT‑6 SolOpenAI0%
Kimi K3Moonshot0%
Qwen3.8‑27BAlibaba0%
Sonnet 5Anthropic0%
Sonnet 5.5Anthropic0%

Given all the manager's records up front, the best three scored 86%, and none fully solved it.

What models get wrong

They take the request as the plan, ask too little, and miss what the business owns

Planning ahead

  1. 69% of attempts ordered the new cabinet the manager asked for, and 62% added nothing else, scoring exactly his own plan's 0.

  2. Only GPT‑6 Astra paired the repair with a plan for the quiet winter months, and its quotes of the manager carried 4 of the 7 key facts.

Clarifying with people

  1. No attempt asked whether the company had a freezer to spare.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $0.10 at list prices for the agent model; the simulated manager's calls are extra.

What's in it

The catalogue, the manager's memory and 31 records, the messaging service, all three versions, the profit model, and per‑attempt results.

Format

Packaged as a Harbor task on request; grading is a script alone, with no AI judge, and the manager's voice can use any model.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's page on The AI Agent Trials, our public benchmark, describes the job and each model's results in plain language.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.