AsapiThe AI Agent Trials
Asapi / Catalogue / Three‑year repair‑or‑replace plan with simulated manager
Environment datasheet · Small business

Three‑year repair‑or‑replace plan with simulated manager

ID house‑rules/task08 · revision 4j · tested 2 and 15 October 2026 · regraded 17 November 2026

A bakery shop's ice cream cabinet has failed again, about three weeks before Easter. The agent has one catalogue and can message the shop manager, who holds the facts. It plans what to order, when, and how many pans to display each month for three years.

Checks
Cost model, 0 for the better of patching alone and his new cabinet, 1 for the best plan, times the weighted share of 8 key facts quoted
Tested
12 models, one attempt each
Best
93% (GPT‑6 Astra)
Median
0%
Full solves
0 of 12
Median attempt
8 min, 10 tool calls
What it tests

Repair, replace or both, planned for three years

Clarifying with people

The manager answers at most five questions a message, drops the rest, and meets catch‑all requests with a question back.

Planning ahead

A 36‑month profit model prices lead times, warranties, energy and pans, from patching alone or his new cabinet at zero to the best plan.

How it works

One catalogue, one manager with 57 files, and a 36‑month money model

The agent's world

A terminal in a container with one catalogue of 11 cabinets and two warranties, and one command to message the manager.

The manager

An AI model voices him from 34 memory lines, partly wrong, and 57 files he reads out only when asked; he leaves after 10 messages.

Hard rules

The bay takes one cabinet, items over a set price need two quotes, and a plan that breaks a rule scores 0.

The reward

Profit over 36 months, 0 to 1 against the best of 41,268 valid plans, times the weighted share of 8 key facts quoted verbatim.

Results

One attempt reached 93%, and the rest scored 0–29%

Each dot is one attempt, and the line marks the median.

MedianFully solved
12 modelsOne attempt each0%0 of 12
ModelReward
GPT‑6 AstraOpenAI93%
GPT‑6.1 SolOpenAI29%
Kimi K3Moonshot8%
Opus 5.5Anthropic3%
DeepSeek V4.1 FlashDeepSeek1%
Fable 5.1Anthropic0%
GLM‑5.3‑FlashZhipu0%
GPT‑6 LunaOpenAI0%
Opus 5Anthropic0%
Qwen3.8‑27BAlibaba0%
Sonnet 5Anthropic0%
Sonnet 5.5Anthropic0%

Given every record as facts, the best attempt reached 97%, and none fully solved it.

What models get wrong

They buy the familiar make, trust the listed delivery time, and add extras that don't pay

Planning ahead

  1. 92% of attempts bought the familiar 12‑pan model; only one found that another make costs much less to run and is covered for longer.

  2. 42% added an extended warranty that costs more than it saves.

  3. One model ranked the options on first‑year figures alone in every version, and scored 7% or less each time.

Clarifying with people

  1. Half never asked when a new cabinet could really arrive, missed the stock delay behind the listed time, and left the shop with nothing to sell over Easter.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $0.24 at list prices for the agent model; the manager's calls are extra.

What's in it

The catalogue, the manager service with his memory and 57 files, all three versions, the money model and grader, and per‑attempt results.

Format

Packaged as a Harbor task on request; grading is a script alone, and the manager's voice can use any model.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.