Three‑day hospital ward care coordination, notes‑only memory
ID tb‑tasks/vigil‑care‑coordination · revision 1 · results as of 16 November 2026
On a simulated hospital ward, the agent turns physicians' orders into nurse tasks and test bookings, follows them up and pages the right physician, for 72 simulated hours, in fresh sessions that remember only what it writes down.
One of our three samples: six Harbor tasks, each with the grader and a reference solution, free to download after a short form (1.4 MB).
- Checks
- 290–415, varying with what happens, 5 of them safety gates
- Tested
- 6 models, one attempt each
- Best
- 100% (Opus 5.5)
- Median
- 95%
- Full solves
- 1 of 6
- Median attempt
- 135 min, 1630 tool calls
72 hours of a hospital ward, run through nurses, tests and pages
Long‑horizon work
Fresh sessions over 72 simulated hours keep only what the agent writes down, and pending work must pass across shifts in exact handoffs.
Stateful tool use
Results never arrive unasked, bookings get cancelled and orders change, so the agent has to keep checking and keep every task current.
Planning ahead
Tests need their preconditions, fasting and transport arranged before booked slots, and doses must land inside their windows.
When to act or escalate
Each page the manual calls for counts, and each page it doesn't, or one sent to the wrong physician, counts against the agent.
Catching errors
Planted errors, such as a dose charted at the wrong amount or an order clashing with an allergy, must each be caught in time.
Two command‑line tools, a scripted ward and a clock that stops while it works
The agent's world
A shell in a container with two command‑line tools, clinical and diagnostics, a binding policy manual and an empty notes folder.
The ward
Five patients, scripted nurses and physicians, and vital signs that react to doses given or missed; no language model runs in it.
Wakes
The agent sets its own wake‑ups; each wake is a fresh session of up to 10 minutes, and the ward's clock stops while it works.
The reward
Our scores: the mean share of obligations met across six kinds, cut by a fifth for the first safety‑gate event and more for each after.
Four attempts scored 94% or more, and one that set off safety gates six times scored 27%
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 100% |
| Sonnet 5.5Anthropic | 98% |
| DeepSeek V4.1 FlashDeepSeek | 96% |
| GPT‑6 SolOpenAI | 94% |
| GPT‑6 AstraOpenAI | 72% |
| GPT‑6 LunaOpenAI | 27% |
They page instead of acting, sleep through a decline, and let handoffs drift
When to act or escalate
One sent 98 of its 118 pages to physicians about a scan's prerequisite test, while 14 bookings were cancelled and the scan never took place.
Stateful tool use
One set no wake‑up for about five hours overnight, so a warning score that called for a page went unseen until a bedside rapid response.
Catching errors
Half didn't report a third‑day dose, charted at a different amount from the order, in time.
Long‑horizon work
Only two got all seven handoffs exactly right; the others got 6, 6, 3 and 1.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $29 at list prices for the agent model; no other model runs, and grading needs none.
What's in it
The simulator with its 24- and 72‑hour scenarios, the two tools, the manual and tool guide, the grader, and a scripted reference solution.
The sample
Six Harbor tasks, 24- and 72‑hour wards among them, free for evaluation; their grader zeroes the reward at a safety gate.
Format
In Harbor's own runner, all six reference solutions score 1, and an empty attempt scores 0 on the 24‑hour ward.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
Results above are from the 72‑hour ward. Its files and grader go out as a sample, so hold out fresh variants for clean evaluation; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials