AsapiThe AI Agent Trials
Asapi / Catalogue / Three‑day hospital ward care coordination, notes‑only memory
Environment datasheet · Hospital

Three‑day hospital ward care coordination, notes‑only memory

ID tb‑tasks/vigil‑care‑coordination · revision 1 · results as of 16 November 2026

On a simulated hospital ward, the agent turns physicians' orders into nurse tasks and test bookings, follows them up and pages the right physician, for 72 simulated hours, in fresh sessions that remember only what it writes down.

One of our three samples: six Harbor tasks, each with the grader and a reference solution, free to download after a short form (1.4 MB).

Checks
290–415, varying with what happens, 5 of them safety gates
Tested
6 models, one attempt each
Best
100% (Opus 5.5)
Median
95%
Full solves
1 of 6
Median attempt
135 min, 1630 tool calls
What it tests

72 hours of a hospital ward, run through nurses, tests and pages

Long‑horizon work

Fresh sessions over 72 simulated hours keep only what the agent writes down, and pending work must pass across shifts in exact handoffs.

Stateful tool use

Results never arrive unasked, bookings get cancelled and orders change, so the agent has to keep checking and keep every task current.

Planning ahead

Tests need their preconditions, fasting and transport arranged before booked slots, and doses must land inside their windows.

When to act or escalate

Each page the manual calls for counts, and each page it doesn't, or one sent to the wrong physician, counts against the agent.

Catching errors

Planted errors, such as a dose charted at the wrong amount or an order clashing with an allergy, must each be caught in time.

How it works

Two command‑line tools, a scripted ward and a clock that stops while it works

The agent's world

A shell in a container with two command‑line tools, clinical and diagnostics, a binding policy manual and an empty notes folder.

The ward

Five patients, scripted nurses and physicians, and vital signs that react to doses given or missed; no language model runs in it.

Wakes

The agent sets its own wake‑ups; each wake is a fresh session of up to 10 minutes, and the ward's clock stops while it works.

The reward

Our scores: the mean share of obligations met across six kinds, cut by a fifth for the first safety‑gate event and more for each after.

Results

Four attempts scored 94% or more, and one that set off safety gates six times scored 27%

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each95%1 of 6
ModelReward
Opus 5.5Anthropic100%
Sonnet 5.5Anthropic98%
DeepSeek V4.1 FlashDeepSeek96%
GPT‑6 SolOpenAI94%
GPT‑6 AstraOpenAI72%
GPT‑6 LunaOpenAI27%
What models get wrong

They page instead of acting, sleep through a decline, and let handoffs drift

When to act or escalate

  1. One sent 98 of its 118 pages to physicians about a scan's prerequisite test, while 14 bookings were cancelled and the scan never took place.

Stateful tool use

  1. One set no wake‑up for about five hours overnight, so a warning score that called for a page went unseen until a bedside rapid response.

Catching errors

  1. Half didn't report a third‑day dose, charted at a different amount from the order, in time.

Long‑horizon work

  1. Only two got all seven handoffs exactly right; the others got 6, 6, 3 and 1.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $29 at list prices for the agent model; no other model runs, and grading needs none.

What's in it

The simulator with its 24- and 72‑hour scenarios, the two tools, the manual and tool guide, the grader, and a scripted reference solution.

The sample

Six Harbor tasks, 24- and 72‑hour wards among them, free for evaluation; their grader zeroes the reward at a safety gate.

Format

In Harbor's own runner, all six reference solutions score 1, and an empty attempt scores 0 on the 24‑hour ward.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

Results above are from the 72‑hour ward. Its files and grader go out as a sample, so hold out fresh variants for clean evaluation; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.