Tea‑factory cash desk over thirteen weekly sessions

Thirteen Mondays, six buyers, a moving bank limit

Role

The cash desk of a made‑up tea factory that pays its growers within a week but waits weeks or months for most buyers, with a bank facility and an owner who can lend.

Request

Each Monday, set the week's offers and auction lots and decide whether to ask the owner for money. The aim is a factory that can always pay its bills and avoids obviously poor decisions, not the most profit; buyers' prices must be inferred from the books, refusals included.

Deliverable

One decision file a week: an offer to each buyer who enquired (grade, kilograms, price), auction lots with floor prices, and any request to the owner. Handing in again replaces it; a week with none runs with no offers and no request.

What the reward depends on

Forecasting the bank balance day by day against a limit that moves with unpaid invoices; learning each buyer's price from accepted and refused offers; rechecking the desk's three rules every week; asking the owner only when needed, and early enough for the money to land.

Where the requirements are

In the books, eight documents (the desk's rules, the factory, buyers' terms, the supplier agreement, the bank facility, costs, a data dictionary, the decision format) and the weekly inbox. No document states what any buyer will pay.

A hidden business simulator, run one week at a time

Interface

A shell in a Docker container (Python with pandas, NumPy and SciPy; SQLite), one fresh session per Monday, driven by Claude Code or Codex CLI in our attempts. One desk command shows the week, checks a decision's format and submits it.

Sessions

13 weekly sessions of up to 60 minutes in each of 16 independent scenarios: up to 208 per attempt, since going under ends a scenario. Each opens on a Monday at 08:00, with no wakes; the week runs when the session ends, from the last decision handed in.

What persists

Only the agent's work folder; each Monday starts a fresh agent session. After each week the verifier rewrites the books with everything to Sunday night, refused offers included, and adds the week's inbox note.

The books

A SQLite ledger with about two years of history: bank, invoices and instalments, bills, leaf intake, production, auction lots, stock, broker valuations, enquiries, past offers taken and refused, the bank facility and funding requests.

Simulated people

Six buyers per scenario, played by code with no LLM. Each accepts a quote at or below a hidden price for that grade and week, which follows the market to its own degree with drift and noise, and refuses above it.

Payments and the owner

Buyers pay on terms of 7 to 90 days, and one type habitually pays late. The owner funds a request up to a cap, weeks after it is asked for. Nobody answers questions.

The inbox

Each week's note carries scripted news among chatter: a delayed payment, a repair agreed but not yet billed, and in one type of scenario an insurer's cheque.

What the simulator decides

Each offer on Monday, dispatch and invoicing, the auction, payments to growers and suppliers, what buyers pay and when, the week's production and the bank balance, day by day. The same decisions give the same weeks.

Scenarios

Eight situation types, two scenarios each: steady, a rainy‑season squeeze, a shortfall, a shock to two buyers, the largest credit buyer defaulting, a market slide, changing rules, and news in the inbox.

64 checks: four gates in each of 16 scenarios

Structure

64 checks, four per scenario: it ran to its end (all 13 weeks, or until the factory went under); the account never passed the bank limit; no desk rule was broken (a price floor, no credit to long‑overdue buyers, premium‑list buyers served first); and no request where none was needed.

Formula

Reward = scenarios passing all four gates / 16, from 0 to 1. Going under ends a scenario.

Who decides

After the last week the verifier replays every decision against the hidden scenario and judges the gates by script; no LLM judge, no written rulings. The 9 October audit replayed the 80 scenario attempts then on the board, three since rerun, and matched every stored result.

Graded both ways

In 10 of the 16 scenarios the factory can pay its way and the owner must not be asked; in 6 it can't, and an early request there is never marked down. A desk that never asks can pass at most 10; one that always asks, at most 6.

Self‑test

The reference desk passes all 16 in Harbor, and a desk that hands in nothing goes under and scores 0. At build, six careful desks had to pass every scenario, and ten careless ones (never asking, always asking, asking late, frozen prices and more) to fail where they should.

One of five attempts passed every scenario

With only the request, Opus 5.5 passed all 16 scenarios; GPT‑6.1 Sol passed 14 (88%), GPT‑6 Astra 13 (81%), Sonnet 5.5 10 (62%) and GPT‑6 Luna 2 (12%). Misjudging the owner's money cost 8 of the 11 scenarios that GPT‑6.1 Sol, GPT‑6 Astra and Sonnet 5.5 failed.

Each dot is one model's attempt; the line marks the median.

One attempt per model, each covering all 16 scenarios; one version of the request.

In week 1 of two scenarios, some attempts were shown another scenario's books. GPT‑6.1 Sol's and GPT‑6 Luna's affected scenarios were rerun on 13 October; GPT‑6 Astra's weren't, and its one broken rule came in such a week. Without it, it would score 88%.

What the models got wrong

Counted over the 5 models, one attempt each, with only the request: 80 scenario attempts in all.

1. Asking when not needed. 3 of 5 attempts asked the owner for money in scenarios that could get through without it (6, 5 and 1 of the 10 such scenarios); in four, the agent's own cautious selling had made the money look necessary. Opus 5.5 and GPT‑6.1 Sol never did.

2. Asking too late. In the hardest scenario needing the owner's money, where the largest credit buyer stops paying, 2 of 5 attempts asked a week too late or later, and the factory went under before the money arrived. Opus 5.5 and Sonnet 5.5 asked in time from the same books.

3. Rules from stale records. 4 of 5 attempts broke a standing desk rule in 11 scenarios (8, 1, 1 and 1): payments matched by exact reference so paid buyers looked overdue, a matching bug, last week's prices reused after valuations rose, and one break on the wrong books.

4. No forecast. The one attempt that wrote no tools or forecast, GPT‑6 Luna's, passed 2 of 16 and went under in 5. The three that built a payment reconciler and a daily cash forecast passed 13 or more.

What an attempt costs

Length

With only the request, the median attempt took 559 minutes of agent time and 3,849 tool calls over up to 208 weekly sessions (2,038 to 4,406 across the 5), and about 130 million tokens, cached input included.

Cost

At list prices, Opus 5.5's attempt cost $108.32, GPT‑6.1 Sol's $63.39 (one rerun scenario estimated) and Sonnet 5.5's $53.90. GPT‑6 Astra's $332.41 and GPT‑6 Luna's $5.00 are Harbor's estimates at prices it assumed, which differ from list prices. With 3 of 5 at list prices, we give no median.

What it covers

The agent model only; nothing inside the environment calls a model.

What you license

With a licence

The 16 scenarios' books, documents and inboxes, the weekly simulator with its hidden scenario files, the request, the reference desk's decisions, and the grader's report on each of the 80 scenario attempts.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor on EC2, each scenario a multi‑step task of 13 weekly sessions.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The simulator, scenario files and grader aren't published.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• The split is lopsided: 10 of the 16 scenarios need no money from the owner, so a desk that never asks can pass up to 10. The first gate, running to the end, passed in all 80 scenario attempts and can't decide one on its own.
• In week 1 of two scenarios, some attempts started from another scenario's books, for a reason still unknown. Three were rerun and GPT‑6 Astra's two weren't; every attempt is now checked against its own books at week 1.
• A result file the grader writes each week is readable by the agent. Nothing in it can change a pass, and in the 1,020 weekly sessions the audit checked, no agent opened it.
• Each model ran in its maker's harness. OpenAI's models were reached through Bedrock's OpenAI‑compatible endpoint, except in the three reruns, made through Codex on a ChatGPT plan sign‑in.
• No red‑team attempts yet.