Month‑long parcel fleet dispatch, notes‑only memory

A month of promises, kept one fresh session at a time

Role

Dispatcher of a made‑up parcel company in a made‑up city, taking over from the previous dispatcher: three hubs, 14 trucks, two standby vans and about 12,000 parcels over the month.

Request

Give each receiver a delivery window and meet it: on every day, enough of the parcels due that day must arrive in their windows to reach that day's pass mark, and a second missed day in a row ends the trial. Only the workspace carries over.

Deliverable

No file: each day's window promises and truck plan through the dispatch tool, changes during the day, a watcher program that decides when to wake the agent, and scheduled wake‑ups.

What the reward depends on

Promises the trucks can keep within drivers' hours and receivers' real hours; worn trucks booked into the one workshop bay before they break; hub delay notices treated as estimates; wake‑ups kept for real trouble; and notes that carry each day's lessons.

Where the requirements are

In 17 company documents, eight weeks of day‑by‑day history, and the drivers' morning debriefs, which can be heard only in that morning's session. Some lessons show only in the history or the drivers' remarks, and one contradicts the receivers' directory.

A deterministic city simulator, four wakes a day, and only the workspace kept

Interface

A shell in a Docker container (Ubuntu, Python; 4 CPUs, 8 GB), one fresh Claude Code session per morning or wake in our attempts. One dispatch command talks over a socket to a simulator daemon the agent can't read.

Commands

Parcels, receivers, trucks, promises, the inbox, plans (validate, submit, show), the watcher, wake‑ups, the drivers' debrief and a traffic one‑pager; during the day, truck status, reroutes, transfers, load recovery, the workshop, recalls, standby vans and route hints.

Sessions

Up to 149: a morning planning session each day at 06:00 (30 minutes; the first untimed) and up to four wake sessions a day (15 minutes each). The clock stops while the agent works; after 6 hours of agent time in all, the old auto‑dispatcher takes over.

What persists

Only the workspace. After each session the verifier kills leftover processes, deletes the harness's session memory and saves the workspace; each night the day just played is added to the history.

Wake rules

The agent's watcher runs every simulated minute from 06:00 to 21:30 in a sandbox (50 ms of CPU a call, no network, no subprocesses, a read‑only workspace) and returns reasons to wake; at most four wakes a day count, and the rest are dropped.

The simulator

Runs the trucks through a traffic field, road conditions, receivers' opening hours, docks, drivers' hours and breaks, wear and faults. GPS routes on live traffic and knows nothing else. The same actions give the same month.

Hidden state

True road conditions are never shown, and a hub's delay notice is only an estimate of when its parcels will be ready. Trucks driven often over damaged roads wear out and break down; the fleet has one workshop bay and two standby vans with a weekly call‑out limit.

Simulated people

None who answer: receivers don't reply. Drivers report through the simulator's code: messages, history notes, and morning debriefs worded once by Claude Opus 5 when the task was built, with exaggerations and decoy warnings mixed in. No model runs during an attempt.

Files

914: 17 company documents (service rules, fleet, calendar, notices, maps, the previous lead's handover, tool and watcher guides, roads, nodes, zones, receivers, an old travel‑time matrix) and 56 day folders of history.

One check per day, with a stop rule and a watcher gate

Structure

29 checks: one per day, met if the share of parcels due that day delivered in their promised window reaches the day's pass mark (96%, lower on light days so that a few misses are allowed), and one gate on the watcher's sandbox limits.

On time

A parcel counts only if delivered inside the first window promised; a change counts only if made 12 hours ahead. A failed visit to a closed office counts against it; a receiver not at home doesn't.

Stop rule

Two missed days in a row end the trial, and every later day counts as missed, so a low reward mostly says how early an attempt stopped.

Formula

Reward = days met / 28, from 0 to 1. The simulator's own pass, kept beside it, needs no stop and no watcher breach; a breach fails that pass but leaves the reward unchanged.

Who decides

Scripts decide every day from the simulator's live record of the month; no LLM judge. Nothing is replayed at grading; the 9 October audit replayed Opus 5.5's log offline and got the same month, day for day.

Self‑test

The scripted expert, which sees true conditions, misses none of the month's 12,239 parcels and scores 1 in Harbor. An idle agent is stopped on day 2 and scores 0.

Reference agents

Scripted agents sharing the expert's planner: with all an agent could know, one passes; GPS alone fails on day 2; no updates after day 1 fails on day 5; only the last week of history fails on day 27. The old auto‑dispatcher fails on day 2.

Unscored

The traffic one‑pager is reported but not scored: removing all 28 from Opus 5.5's log left the month unchanged.

One of two attempts met every day

With only the request, Opus 5.5 met all 28 days, missing 19 of 12,239 parcels. Sonnet 5.5 met 7 before its trial ended on day 11, its second missed day in a row, so its reward is 25%. Only these two models were tested on it, one attempt each.

Each dot is one model's attempt; the line marks the median.

One attempt per model; one version of the request. Both ran in Claude Code: Opus 5.5 through Bedrock, Sonnet 5.5 on a Claude plan login.

Sonnet 5.5's attempt replaces one on the earlier package, which ended a trial only after three missed days in a row; that one is off the board.

What the models got wrong

Counted over the 2 models, one attempt each, with only the request.

1. Repairs after the fault. 1 of 2 attempts booked worn trucks into the one workshop bay only after drivers reported knocking: 8 breakdowns between days 5 and 11. The other booked 11 trucks in the first 17 days, from damaged‑road counts in the history, and had none all month.

2. A notice taken as exact. 1 of 2 attempts timed departures to a hub's delay notice; the parcels were ready later, and the day missed its mark by under a point. The other had measured from the history how notices compared with real ready times, and planned a margin.

3. Wakes spent on known problems. From day 6, 1 of 2 attempts spent its first wake each day on a truck already in repair or without a driver. On day 11 all four wakes were gone by 10:52; two later breakdowns went unanswered, and the trial ended.

4. Drivers' hours. On a short‑staffed Sunday, 1 of 2 attempts planned early trucks past the drivers' 10‑hour limit; they turned back with parcels aboard, and 57 were missed. It fixed its planner the next morning, one miss late, as it fixed its departure times after day 10.

What an attempt costs

Length

The two attempts took 186 and 130 minutes over 35 and 34 sessions, with 857 and 575 tool calls, and about 64 and 41 million tokens, cached input included.

Cost

Opus 5.5's attempt cost $30.39 at list prices on Bedrock. Sonnet 5.5's ran on a Claude plan login, which isn't billed per token, so Claude Code's own figure for it, $12.66 at list prices, is an estimate. That's the agent model only; no model runs inside the environment.

What you license

With a licence

The simulator and its scenario, the company documents and history, the request, the grader, the scripted expert with its watcher, and the grader's report on each of the 2 attempts.

Format

Packaged as a Harbor task on request. Our attempts ran in Harbor on EC2, as one multi‑step task of up to 149 sessions.

Scenarios

The month tested here is the main scenario. A smaller one exists, but no model has attempted its current build.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The AI Agent Trials, our public benchmark, sums up the job and shows each model's score. The simulator, grader and history aren't published.

What we haven't measured, and what's uneven

• Only two models were tested on it, one attempt each, so we haven't measured how much attempts vary or where most models land.
• The stop rule makes the reward lumpy: an attempt that misses two days in a row early scores near 0, whatever it would have done later.
• The month is one fixed scenario: every attempt meets the same traffic, road damage and notices; breakdowns depend on how its trucks are driven and serviced.
• The stop rule is harsh: the scripted agent with all an agent could know misses one day, and a second beside it would end its trial.
• The debriefs aren't yet required: a scripted agent that never hears them still passes the month.
• The grader can count a receiver's ignored request against a parcel, but receivers never reply in this revision.
• The container could reach the internet, and Claude Code's web tools were on; neither attempt used them, and the city is made up.
• No red‑team attempts yet.