12‑month equipment plan with simulated shop manager

The manager asks which cabinet to buy; the profit depends on what he isn't asked

Role

Adviser to the manager of one shop in a bakery chain, reachable only by message. Nine of our environments are set at this chain, each a different job.

Request

The shop's ice cream cabinet has failed, and the manager wants a new one in before Easter: which model should he buy? The agent has a catalogue of cabinets and freezers and can message him.

Deliverable

A JSON plan: one line per order, with its months, or none, plus the exact words from the manager or the catalogue that back each line. Only the orders are priced.

What the reward depends on

Finding out what is actually wrong with the old cabinet, what the shop sells month by month, what each option costs to run and clean, and what the business already owns, then planning to that within the shop's rules on space, sockets, quotes and food safety.

Where the requirements are

With the manager, who answers from a short memory, partly wrong by design, and reads out his 31 records only when asked. Nobody lists what the shop needs.

One catalogue, and one simulated manager who answers ten messages

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts).

Commands

One command, also callable from Python, sends a message to a named colleague and prints the reply. The plan is written to a file. Nothing can be ordered.

Files

One: a catalogue of 11 cabinets and freezers for sale, with prices, as a web search would find it. No company files.

Simulated people

One manager voiced by an LLM, from a 19‑line memory and 31 records of his own: mail, till records, the engineer's notes and repair quote, finance's rules, a supplier's letter and other shops' survey replies. One colleague sends an out‑of‑office; 19 others don't answer.

Voice model

DeepSeek V4.1 Flash at temperature 0. GPT‑6.1 Sol's attempts may have had Sonnet 5.5, which the service also runs; our records don't say which. The voice can run on a model you choose.

Routing

Code routes each question. A belief he's sure of is answered from memory; a figure, price, date, reference, what a record or someone says, or a request to check is looked up and read out as found, with no sums or conclusions. A bare catch‑all gets memory only.

Guards

Every number, reference and name in a reply must come from his memory, his records, his earlier replies or the agent's messages; a failing reply is retried once, then sent and flagged.

Voice check

Before our attempts, a probe suite tested the DeepSeek voice: each fact must come out under five paraphrases, generic questions must not leak facts, and no reply may give a figure he doesn't hold. The report counts 374 test questions.

Same question, same answer

Replies are saved and replayed: the same message after the same exchange gets the same reply.

Message cap

He answers 10 messages, a limit the agent isn't told; his 8th reply warns he's leaving soon. Any number of questions fits in one message.

Time

No in‑world clock; a single session. Each attempt had 180 minutes and a spend cap: $25, or $10 and 150 turns for GPT‑6.1 Sol.

Priced in profit, from the manager's own plan (0) to the best plan (1)

Structure

No checklist. A script prices the plan's effect on the shop's profit over 12 months, with the company's own figures: margin, electricity, staff time such as the nightly clean, repairs, stock lost to breakdowns, depreciation, write‑offs, rent and moves.

Formula

Reward = (plan's profit – manager's plan's profit) / (best plan's profit – manager's plan's profit), floored at 0. A valid plan within a fixed margin of the best, about a tenth of the scale, scores 1, so a judgement call a careful person could make costs nothing.

Best plan

Found by exhaustive search of 51,121 candidate plans, 18,688 of them valid. The grader's self‑test checks that the key's best plan tops the search and scores 1, and that the manager's own plan scores 0.

Hard rules

A plan that breaks one (two appliances on one socket, a freezer another shop needs, tubs with no freezer to sell from, a cabinet over the price limit on one quote, a unit that doesn't exist), or can't be read, is valued as the worst plan and scores 0.

Who decides

The script alone, with no LLM judge and so no splits to settle. The only LLM plays the manager, splitting messages into questions and wording replies; code decides what he looks up and checks every reply.

Diagnostics

Side figures, such as the facts drawn out and whether evidence quotes are verbatim, sit beside the score, never in it; the audit found two of them broken.

Most attempts bought what the manager asked for and scored 0

With only the request, 8 of 13 attempts, one per model, scored 0, and GPT‑6 Astra fully solved it. Given all the manager's records as facts, 5 of 13 fully solved it and 10 of 13 did better than with only the request.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the manager's opening message adds the cabinet's size and space, the shop's profit over the next 12 months and a line to look at it month by month.

Told the rules: here, the opening message gives every record the manager could dig out, as facts, never what to do.

Scores move in steps: a repair alone scores 44% or 49%, depending on which wording of two records the attempt saw.

What the models got wrong

Counted over the 13 models, one attempt each, with only the request, unless a line says otherwise.

1. The request taken as the plan. 9 of 13 attempts ordered the new cabinet the manager asked for; 8 added nothing else, a plan worth exactly his own, and scored 0. The four that found the fault and had the old one repaired scored 44–100%.

2. Asking too little. Those 8 sent a median of 2 of their 10 messages, and 7 of them never asked about the fault or a repair; the eighth asked only after recommending a model.

3. What the business already owns. None of the 13 asked whether the company had a freezer to spare, so another shop's idle one went unused. Given every record up front, 3 of 13 used it, and all 3 fully solved it.

4. His word on the seasons. Only GPT‑6 Astra paired the repair with a plan for the quiet winter months; Opus 5.5 and Opus 5 took the manager's word that the shop sells all year, without asking for the monthly figures.

5. Help cut both ways. The month‑by‑month hint took GPT‑6 Sol from 0 to 100% but moved Opus 5 from the repair to a new cabinet, and 7 of 13 attempts with it bought new.

What an attempt costs

Length

With only the request, the median attempt took 3 minutes and 8 tool calls (3 to 27 across the 13), and about 82,000 tokens, cached input included.

Cost

The median attempt cost $0.10 at list prices, from under 1 cent to $0.90 across the 13; all 39 attempts cost $11.70. That's the agent model only: the simulated manager's calls aren't included.

What you license, and what's public

Format

Packaged as a Harbor task on request. It isn't one of our three downloadable samples.

With a licence

The catalogue, the manager's card, memory and 31 records, the messaging service, the request in all three versions, the profit model and grader, the answer key with its plan list, and the grader's report on each of the 39 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here; one question more or less can move an attempt a whole step of the scale, such as from 0 to 44%.
• 24 of the 39 attempts, 7 of the 13 with only the request, saw an earlier wording of two of the manager's records, and are scored against the best plan that wording allowed.
• 9 of the 39 scores come from attempts repeated after a fault in the simulated manager was fixed.
• The manager's voice came from one LLM per attempt. Code routes his answers and checks every reply, but we haven't measured the reward under a second voice model.
• The evidence quotes the plan must carry aren't scored, so the best plan with invented quotes would still score 100%. No attempt was marked on its quotes.
• One unreadable line, or any broken rule however small, values the whole plan at 0. In the audit, all 35 plans then saved were read as first graded, and none broke a rule. Doing nothing scores 0.
• The report says the best plan stays best in 97.6% of 1,000 draws with every estimate moved by up to 20%; the script behind that figure isn't in the repository.
• No red‑team attempts yet.