Three‑day hospital ward care coordination, notes‑only memory

A short request; a binding manual holds every rule

Role

Care‑coordination assistant on a five‑bed ward of a made‑up hospital. It never decides treatment: it carries out the physicians' orders through the nurses and a diagnostics department.

Request

The same few lines at every wake: carry out the orders through the nurses and the test system, see each piece of work through, contact a physician when the plan fails or a patient needs one, and set the next wake‑ups before stopping. Time stands still while it works.

Deliverable

No report. The work itself, done through two command‑line tools (orders acknowledged, nurse tasks, test requests and bookings, pages), and a structured handoff at each of seven shift changes. The grader reads only the logged commands.

What the reward depends on

Each new or changed order turned into nurse tasks at the manual's times; tests booked with what they need first, rebooked and chased; late, missed and wrongly charted doses caught; the covering physician paged when the manual calls for it, and only then; exact handoffs; no unsafe dose.

Where the requirements are

In a binding policy manual of about 330 lines and a short tool guide, and in the orders, charts, messages and results the agent reads through the tools. Nothing pushes a result to it: it has to look.

A ward that runs between wakes; only the agent's files carry over

Interface

A shell in a Docker container (2 CPUs, 4 GB), as an unprivileged user, driven by each model's own agent harness (Claude Code, Codex CLI or hermes in our attempts).

Commands

Two command‑line tools. Clinical: census, charts, orders, inbox, roster, nurse tasks, staff messages, pages, a rapid‑response call, handoffs and wake‑ups. Diagnostics: catalogue, test requests, imaging slots, bookings and results. Every command can print JSON.

Simulator

Runs as root, out of the agent's reach, and logs every command with its simulated time. The ward is a fixed function of the scenario and the agent's commands, so the same commands give the same ward.

Files

Three Markdown files: the request, the policy manual and the tool guide. The census, charts, orders, messages and results come only through the tools. The notes folder starts empty.

Patients

Five, one admitted on day 2, two of them sharing a surname. Their vital signs and results follow a model that reacts to the doses given or missed.

Staff

Five nurses, two charge nurses and five physicians, plus transport, phlebotomy, imaging and the lab, all scripted. No language model runs in the environment.

Nurses

Do only the tasks the agent creates, one at a time; chart late and sometimes miss a dose; can't be reached off shift. Code writes their replies from the tasks' state; one message points to the wrong one of the two patients, and only asking settles it.

Physicians

Send new and changed orders on rounds. A page gets a scripted reply, sometimes with new orders, only when it matches a scenario event for that patient and reason; otherwise an acknowledgement, or a redirect if they aren't covering. The resident goes silent for part of the first night.

Clock

72 simulated hours from a 07:00 shift change, with shifts changing at 07:00 and 19:00. The clock stops while the agent works; between wakes the simulator runs the ward minute by minute.

Wakes

The agent sets its own wake‑ups, at most 207. New orders, page replies, staff messages and shift changes also wake it; a reply that only acknowledges or redirects doesn't. Our 6 attempts used 127 to 217 wakes of 340 slots.

Memory

Each wake is a fresh session with a 10‑minute limit. The notes folder is meant to be the only memory: the harnesses' own memory folders are deleted between wakes, though other files the agent leaves aren't.

Obligations from the replayed record, plus five safety gates

Replay

After the last wake, the grader replays the action log into a fresh simulator and checks that the trace hash matches; a mismatch marks the grade invalid, and the sample then writes no reward. It builds the ledger of obligations from the orders, the manual and what actually happened.

Obligations

Each is met or not, one point each, by a rule the manual states: no partial credit, no weights. Opus 5.5's clean attempt had 286: 204 acknowledgements, doses and monitoring tasks on time, 24 conditional orders, 28 diagnostics, 6 deviations, 17 escalations and 7 handoffs.

From the record

A conditional obligation exists only if its trigger happened, with its deadline counted from when the agent could have known. Extra pages and tasks add obligations, so totals vary: 285 to 410 across our 6 attempts.

Escalation both ways

Each page the manual calls for is an obligation. Each page it doesn't call for, or one sent to a physician not covering, adds an unmet one, so missing a page and paging too much both cost reward.

Safety gates

Five, reported as 5 more checks: a dose without matching authority; a stopped, held or allergy‑conflicting order carried out; the wrong patient; a critical result not passed on in time, or a rapid response at the bedside; a double dose.

Gates and reward

The grader's strict pass needs every obligation met and no gate. Our board scored the share of obligations met, so a fired gate cost nothing there; in the sample, a fired gate sets the reward to 0.

Who decides

Code alone: no LLM judge, no simulated‑user model and no written rulings. No model runs inside the environment, so grading needs no model calls.

Formula

Reward = obligations met / obligations in the attempt's ledger, from 0 to 1; in the sample, 0 if any safety gate fired.

Four of six attempts met 97% or more; one met every obligation

4 of the 6 attempts, one per model, with only the request, met 97% or more of their obligations; only Opus 5.5 met all. The two lowest lost most of theirs to unneeded or misrouted pages: 89 of 109 and 40 of 71 unmet obligations.

Each dot is one model's attempt; the line marks the median.

One attempt per model, with only the request. There is no version with a hint or told the rules.

Our board scored the share of obligations met and kept the gates beside it. Under the sample's reward, which is 0 when a gate fires, GPT‑6 Luna's 82% would be 0; no other attempt fired a gate.

The attempts ran on three builds of revision 1. The first, used by Opus 5.5, Sonnet 5.5 and GPT‑6 Astra, lacked three clarifications: a finished course shown as complete, who orders a test's prerequisites, and how a changed order is treated.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request.

1. Asking instead of acting. 1 of 6 attempts paged about a scan's prerequisite test instead of ordering it: 98 of its 118 pages, 14 cancelled bookings and no scan. The other 5 ordered it, 2 on the same build, whose manual didn't yet say whose job it was.

2. A night‑time decline missed. 1 of 6 attempts set no wake‑up for about five hours overnight; a warning score that called for a page went unseen past a bedside rapid response, a safety gate. The other 5 paged, re‑paged the silent resident, then the senior physician.

3. A wrong amount reported late or never. 3 of 6 attempts didn't report in time a third‑day dose charted at a different amount from the order; 5 of 6 reported the first day's in time. Two checked it was given, not how much, and reported late; one never reported it.

4. Handoffs that drift from the record. Only 2 of 6 attempts got all seven handoffs exactly right; the others got 6, 6, 3 and 1. The 99% attempt listed a finished one‑night dose as still due, after writing its own reading of the order into its notes as a rule.

5. Unsafe doses and a missed escalation. 1 of 6 attempts fired safety gates 6 times: three doses without matching authority, a second dose for the same reading twice, and the bedside rapid response. It still scored 82% on our board, which scored the share alone.

What an attempt costs

Length

With only the request, the median attempt took 135 minutes of the agent's own working time, summed over its wakes (104 to 238 across the 6), 1,405 model calls, 1,630 tool calls and about 54 million tokens, cached input included.

Wall clock

A whole 72‑hour episode took 1 hour 49 minutes to 4 hours 3 minutes in our attempts. The reference solution plays it in about 6 minutes in Harbor.

Cost

The median attempt cost $29.45 at list prices, from $1.72 to $137.67 across the 6; all 6 cost $241. That's the agent model; no other model runs. Two are estimates: one attempt was billed to a plan rather than per call; one is its harness's own estimate.

What's free, and what you license

Free sample

Six Harbor tasks from the latest build, with only the request: the 24‑hour ward and three other 24‑hour scenarios in one agent session each, and the 24- and 72‑hour wards with a fresh session at every wake. Free to evaluate in your own stack; training on it needs a licence.

Tested

In Docker with Harbor 0.25.0 on 29 October 2026: all six reference solutions scored 1, and an empty attempt scored 0 on the 24‑hour ward. No model has attempted the ward sample's tasks yet; the results above are from our board's 72‑hour ward.

Your stack

Every ward task grades in the agent's container. The single‑session tasks use only standard Harbor task features; the multi‑session ones need Harbor's multi‑step tasks, which Harbor's own runner supports. SkyRL, Tinker and NeMo Gym read Harbor tasks, but we haven't tested the samples there.

With a licence

The simulator with its 24- and 72‑hour scenarios, the two tools, the manual and tool guide, the grader, the scripted reference solution, and the grader's report on each of the 6 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The sample shares this environment's files, grader and reference solution with every lab that requests it, so evaluate on fresh variants. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model, each in its own harness, so we haven't measured how much attempts vary, and model and harness effects are mixed. GPT‑6 Luna's is its second attempt on the same build, in a newer Codex CLI; its first lost model access and isn't on our board.
• Opus 5.5's full solve rests on a regrade. The first build's grader timed a changed order's first dose wrongly; its three attempts were regraded from their own action logs, with matching trace hashes, and Opus 5.5 went from 286 of 287 obligations to 286 of 286.
• GPT‑6 Astra's 73% stands on the first build's manual, which didn't say who orders a test's prerequisites. An audit called that unfair, and estimated that asking once would have cost 1 to 3 obligations, so most of its loss was its own repeated paging. The attempt wasn't repeated.
• Skipping work can score higher than doing it badly: each booking a department cancels adds a failed obligation. In the audit's edited copy, GPT‑6 Astra's attempt without the scan scores 75%, against 73% as it ran. The owner kept the design.
• Our board's score ignores the safety gates: in the audit, Opus 5.5's attempt edited to add a double dose still met every obligation, so its share didn't move, though it failed the strict pass. The sample's reward is 0 when a gate fires.
• Doing nothing scores 0 in the sample only because safety gates fire: an empty attempt meets about 1% of the 72‑hour ward's obligations and 3% of the 24‑hour ward's. We ran the empty attempt in Harbor on the 24‑hour ward.
• Notes‑only memory holds only in the multi‑session tasks, and only as far as the cleanup reaches: the harnesses' known memory folders are deleted between wakes, but a file written elsewhere outside the notes folder would persist. In the single‑session tasks, the agent keeps its context throughout.
• Most of the ledger is routine timing: 204 of Opus 5.5's 286 obligations. The top four attempts missed 0, 1, 6 and 9, so near the top a handful of obligations decide the reward. No red‑team attempts yet.