Three‑year repair‑or‑replace plan with simulated manager

The request asks what to do; the facts come only from asking

Role

Adviser to a bakery chain's shop manager, by message from a terminal.

Request

The manager writes that the shop's ice cream cabinet has failed again and won't hold its temperature. He's thinking of a new one but doesn't know if it's worth it, and asks what to do.

Deliverable

A plan in a fixed format: what to order and in which month, how many pans to display each month for three years, the quotes that back it, and an optional note that is never scored.

What the reward depends on

Seeing past the first answers: a listed delivery time that hides a stock delay, a familiar make that costs more to run, an engineer's advice read too widely, a colleague's sheet built on the wrong figures, and no stated horizon.

Where the requirements are

With the manager: in his memory, partly wrong by design, and in 57 files of his own that he reads out only when asked. The catalogue lists prices, sizes and delivery times, nothing on running costs or stock.

One simulated manager holds the facts, for ten messages

Interface

A terminal in a Docker container, driven by each model's own agent harness (Claude Code, Codex CLI, Kimi CLI, Prime Agent or Qwen Code in our attempts). Web search and fetch are off.

Commands

One command, or a Python call, sends a message to a named person and prints the reply. Nothing can be ordered or paid: the grader prices the plan.

Files

One catalogue of 11 cabinets and 2 extended warranties, with prices, sizes and listed delivery times, standing in for a web search.

Simulated person

The shop manager, voiced by an LLM from a persona, 34 memory lines and 57 files of his own (27 emails, 16 PDFs, 14 text, Word and CSV files) that the agent never sees. Messages to 20 other named staff and suppliers go unanswered.

How questions are answered

An LLM call splits each message into questions and matches them to memory lines. Code routes each one: a line he is sure of is answered from memory, anything else looked up in his files, with up to 8 tool calls a reply.

Limits on asking

He answers at most 5 questions a message and says nothing of the rest. Asked for everything, he asks which ones; asked only whether there's anything else, he answers from memory, with no lookups. He leaves after 10 messages, warning with the 8th; the agent isn't told.

Guards

Every number, ID and name in a reply must come from his memory, his files, his earlier replies or the agent's messages, and code checks that a lookup was made. A failing reply is retried once, then cut or flagged.

Voice model

Sonnet 5.5 in our attempts, on Amazon Bedrock with thinking at its lowest setting; the route for GPT‑6 Luna's three attempts isn't recorded. It can run on a model you choose.

Same question, same answer

Our board saved routes and replies and replayed them across attempts: the same message after the same exchange got the same reply.

Time

No clock. The prompt fixes the date, about three weeks before Easter.

Priced in money, from patching alone to the best plan

Structure

No checklist. A money model prices the plan's effect on the shop's profit over 36 months: margin on sales, electricity, staff time, repairs and call‑outs, stock lost in failures, days shut, depreciation and write‑offs.

Scale

0 is patching the old cabinet and nothing else; worse plans are floored at 0. 1 is the best plan in a search of 41,268 valid plans, recorded in the answer key; an audit's search of 4,519 order sets found no better orders. There's no tolerance band: every pound counts.

Hard rules

Breaking one scores 0: the bay takes one cabinet, which must fit; items over a set price need two quotes; nothing is done to the old cabinet once it's replaced; a buy‑back needs a new cabinet. A line the grader can't read is skipped and listed.

Horizon

Nobody states one. The best plan stays best over any horizon from 15 to 36 months; judged over a single year, patching alone wins.

Who decides

A script alone, with no LLM judge. The manager's log of questions and files read feeds diagnostics, never the reward.

Calibration

Buying only the cabinet the manager has in mind scores 56%. In the grader's self‑test, the best plan scores 1 and changing nothing scores 0.

Formula

Reward = max(0, (U(plan) − U(patch only)) / (U(best) − U(patch only))), where U is the plan's effect on profit over 36 months.

No attempt found the best plan, though six scored 99%

Every attempt saved a plan the grader could read. With only the request, 11 of 12 scored 52–82%: buying only the familiar cabinet scores 56%. No attempt in any version found the best plan; told the rules, 6 of 12 came within 5 points.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the manager's opening message also gives the cabinet's size and the space for it, and says he must watch the shop's profit beyond this year.

Told the rules: here, the opening message also carries the hint's lines and every record the manager could dig out, as facts; it never says what to do.

What the models got wrong

Counted over the 12 models, one attempt each, with only the request, unless a line says otherwise.

1. The familiar make. 11 of 12 attempts bought the 12‑pan model the manager had in mind. Only one found that another 12‑pan make, listed by price and size alone, costs much less to run and is covered for longer.

2. The listing trusted. 6 of 12 never asked when a new cabinet could really arrive, missed the stock delay behind the listed delivery time, ordered no stop‑gap repair and left the shop with no ice cream to sell over Easter. The other six patched the old cabinet.

3. Extras that don't pay. 5 of 12 added an extended warranty that costs more than it saves. 4 of 12, the third‑best attempt among them, kept every pan on display in every month for three years.

4. The short view. 1 of 12 models ranked the options on the first year's figures in all three versions, scoring 59–63% each time. With the hint about later years, another took a colleague's one‑year sheet at its word and patched alone, scoring 0%.

5. Help didn't close the gap. Told the rules, with every record given as facts, 6 of 12 came within 5 points of the best plan. Four of the other six still ordered no stop‑gap, one with a smaller cabinet; the other two left out the buy‑back.

What an attempt costs

Length

With only the request, the median attempt took 8 minutes and 10 tool calls (4 to 47 across the 12), and about 0.19 million tokens, cached input included.

Cost

The median attempt cost $0.24 at list prices, from half a cent to $1.85 across the 12; all 36 attempts cost $25. That's the agent model only: the manager's calls aren't included. GPT‑6 Luna's cost uses a default price, as it has no published one.

What you license

With a licence

Revision 4j in all three versions: the catalogue, the manager service with his persona, memory and 57 files, the money model and grader, the answer key, and the grader's report on each of the 36 attempts, as baseline results.

Format

Packaged as a Harbor task on request. Today it runs in Docker, with a standard‑library Python harness, grader and manager service; the manager's model is reached on Amazon Bedrock or through the Claude Code CLI.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The files, grader and answer key aren't published.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• The manager's voice came from one LLM. Code routes questions and checks replies, but we haven't measured the reward under a second voice model.
• The agent can read and change the label its messages carry. A new label would give it ten fresh messages, or the version told the rules. None of our attempts used it; one printed the label and sent a test message under another.
• A full solve needs a detail worth about 0.3 points: selling packed tubs while the patch is awaited. No attempt in any version did it.
• The plan reader was loosened on 2 October, after 33 of our attempts, to accept lists of months, which the prompt's examples don't show. Under the reader as it was, four plans would have scored 0, the best attempt with only the request among them.
• An audit on 9 October regraded the 33 plans then on the board and changed no score.
• No red‑team attempts yet.