The ward coordinator

Coordinate a hospital ward for three simulated days over more than a hundred fresh sessions with only notes for memory, carrying out orders and paging a doctor only when needed.

Setting: Hospital ward, care coordination
Deliverable: Three days of ward work, done through the hospital's tools

Long-horizon judgementConflicting sourcesAsking the right questionsData quality

Models

ran the standard setup

100%

Best score

Opus 5.5

98%

Median score

across those models

2.3 h

Typical run

median time, standard setup

The task

The agent is the care-coordination assistant on a five-bed ward of a made-up hospital, for 72 simulated hours. It does not decide treatment: it turns the physicians' free-text orders into nurse tasks and tests, follows them up, and pages the right physician when the plan stops working. It has two command-line tools, a policy manual and a tool guide. Every wake is a fresh session, and only its own notes carry over. Simulated nurses, physicians and patients react to what it does.

The prompt only asks it to carry out the orders and follow up. The real test is what a careful ward coordinator does unprompted: keeping exact track of doses, tapers and pending results across sessions it will not remember, reading the record rather than trusting a summary, and knowing when a physician needs to hear and when not.

What it tests

Judging over the long run

Doses, holds and pending results stretch across days and dozens of sessions, and only the agent's own notes connect them.

Trusting the record over a summary

A task list, a nurse's message and the agent's own notes can all disagree with what the chart and the orders say.

Knowing when to ask

A needed page that is missed counts against the agent, and so does one that was not needed or went to the wrong physician.

Catching what goes wrong on the floor

Late charting, missed doses and readings close to a limit have to be noticed and handled.

Challenges

1No memory

Every session starts fresh, so anything not written down is gone, much as at a real shift change.

2Nurses who are human

They chart late, miss doses and finish shifts with work left over.

3Plans that change daily

Morning rounds rewrite the treatment plans, so old orders stop and new ones start.

4Readings near the line

Some readings fall just short of a trigger, and paging on them is a false alarm.

5Results are never sent

The agent learns of a result only by looking for it.

6Names that are easy to confuse

Two patients share a surname, and a message can name the wrong bed.

Results

Model	Standard	Time (standard)	Cost (standard)
Opus 5.5	100%	106 min	$41.16
Sonnet 5.5	99.7%	125 min	$40.07
GPT-6 Sol	98%	104 min	$18.83
DeepSeek V4.1 Flash	97%	146 min	$1.72
GPT-6 Luna	82%	238 min	$1.74
GPT-6 Astra	73%	157 min	$137.67

One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

Opus 5.5 was the only model to run the ward for three days without a miss (100.0%). It kept one running log and ended every wake with an exact plan for the next.
Sonnet 5.5 (99.7%) missed one obligation, in a single handoff: it had written its own guess about an order into its notes as a rule.
The commonest failure was trusting a summary over the record. GPT-6 Sol (98.0%), DeepSeek V4.1 Flash (97.0%) and Sonnet 5.5 each lost obligations by relying on a task list, a note or a booking instead of what the chart and the orders said.
GPT-6 Astra (73.4%) did the bedside work without fault, but treated a test that a scan needed as something only a physician could order, and re-paged the physicians about the scan for three days. It was also the most expensive run, at $137.67.
GPT-6 Luna (82.4%) was the only run with unsafe acts, 6 of them, mostly insulin given twice for one reading. It missed a patient's overnight deterioration, then spent most of its remaining wake-ups paging about her while she was in intensive care, and ran out of wake-ups late on the last evening.
DeepSeek V4.1 Flash met 97.0% of its obligations for an estimated $1.72.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Simulated hospital ward: Ward 4B of St. Aldric Hospital, a made-up hospital, with four patients and a fifth admitted on day 2, run for 72 simulated hours by a deterministic simulator in the container that the agent cannot read
Agent's interface: A terminal in a fresh Docker container (Ubuntu 24.04, Python 3; 2 CPUs, 4 GB), one fresh session per wake as an unprivileged user, driven by the agent's own command-line tool through harbor: Claude Code, Codex CLI or hermes
Documents: 3 Markdown files in /docs: the instruction, the hospital policy manual and the tool guide. The notes folder /workspace starts empty. Everything else (census, charts, orders, messages, results) comes through the two tools
Applications: Two commands: ward (census, charts, orders, inbox, roster, nurse tasks, staff messages, pages, rapid response, handoffs, wake-ups) and dx (test catalogue, test orders, imaging slots and bookings, results). Every command can print JSON. No other apps
Web: Not needed: nothing the task asks about is online. The container can reach the internet. Claude Code (Opus 5.5, Sonnet 5.5) had its web page reader (WebFetch) on and no web search; Codex CLI (GPT-6 Sol, GPT-6 Astra, GPT-6 Luna) had web search turned off; hermes (DeepSeek V4.1 Flash) had only its terminal and file tools. No run used a web tool.
People: Played by the simulator's code, with no language model: 5 ward nurses (3 on days, 2 on nights) and 2 charge nurses, who do only the tasks they are given and answer messages from the state of those tasks; 5 physicians (3 attendings, a resident and an on-call attending), who write orders and answer pages with scripted replies; transport, phlebotomy, imaging and the lab; 5 patients whose vital signs and results follow what is done
Time: A simulated clock from Thursday 5 March 2026, 07:00, for 72 hours. It stops while the agent works; between wakes the simulator runs the ward minute by minute. Shifts change at 07:00 and 19:00
Actions: Acknowledge orders; create, cancel, reassign and reschedule nurse tasks; order, book and cancel tests; message staff; page physicians; call the rapid response team; submit handoffs; set wake-ups. The agent cannot prescribe: every nurse task and test names the order it carries out
Deliverable: No file: the ward's work, done through the two tools, and a JSON handoff submitted at every shift change (ward handoff submit FILE)

← PreviousThe radiation-dose registry All 34 tasks Next →The parcel dispatcher