Month‑long parcel fleet dispatch, notes‑only memory
ID tb‑tasks/kestrel‑fleet‑dispatch · revision 1 · results as of 30 October 2026
As dispatcher of a simulated city parcel carrier for 28 days, the agent promises every receiver a two‑hour window, plans each day's trucks and answers breakdowns and delays through a watcher program and wake‑ups, remembering nothing between sessions but its own files.
- Checks
- 29, one a day, plus a gate on the watcher's sandbox limits
- Tested
- 4 models, one attempt each
- Best
- 100% (GPT‑6.1 Sol and Opus 5.5)
- Median
- 62%
- Full solves
- 2 of 4
- Median attempt
- 158 min, 716 tool calls
A parcel fleet dispatched for 28 days, remembering only its own files
Long‑horizon work
Fresh sessions over a simulated month keep only the agent's files, and each day's promises and truck wear carry into the next.
Stateful tool use
Promises, truck wear and repairs carry over day to day, so each day's plan has to start from what's actually due.
Planning ahead
Worn trucks need the single workshop bay before they break, and each day's promised windows face a pass mark.
When to act or escalate
Wake‑ups are capped at four a day, so a watcher that fires on known or harmless events leaves later breakdowns unanswered.
A city simulator, 14 trucks, a watcher program and four wake‑ups a day
The agent's world
A shell in a container with one dispatch command, covering parcels, trucks, plans, the watcher and wake‑ups, to a simulator it can't inspect.
Sessions
A morning planning session each day and up to four wakes; the clock stops while the agent works, and only its files carry over.
Hidden state
True road conditions are never shown, a hub's delay notice is only an estimate, and trucks wear out on damaged roads.
The reward
One check a day, met when 96% of its parcels arrive on time; two missed days in a row end the trial.
Two attempts met every day of the month; the other two were stopped on days 3 and 11
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| GPT‑6.1 SolOpenAI | 100% |
| Opus 5.5Anthropic | 100% |
| Sonnet 5.5Anthropic | 25% |
| GPT‑6 LunaOpenAI | 4% |
They drop promised parcels, repair trucks too late, and spend wakes on known problems
Long‑horizon work
One planned its second day from new arrivals alone, leaving off parcels it had promised, then planned day 3 with no travel times and was stopped.
Planning ahead
One booked worn trucks into the workshop only after drivers reported knocking, and had 8 breakdowns between days 5 and 11.
One planned early trucks past the drivers' 10‑hour limit on a short‑staffed Sunday; they turned back with parcels aboard.
When to act or escalate
From day 6, one spent its first wake each day on a truck already in repair or without a driver; on day 11 all four wakes were gone by 10:52, two later breakdowns went unanswered, and the trial ended.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.22 to $30.39 for the agent model; no model runs inside the environment.
What's in it
The simulator and its scenario, the company documents and history, the grader, and a scripted expert with its watcher.
Format
Our attempts ran in Harbor as one multi‑step task, a session per morning or wake; other packaging on request.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials