Each sample is a folder of Harbor tasks: the container, the request, the grader and a reference solution. Fill in the short form below to download them. We've run them in Harbor's own runner; SkyRL, Tinker and NeMo Gym read Harbor tasks too, but we haven't tested them there.
Hospital ward coordination
Coordinate care on a simulated hospital ward: carry out physicians' orders through scripted staff, follow up, and page the right doctor when the plan stops working.
- Tasks
- 6: a 24‑hour ward with 4 patients, three other 24‑hour scenarios, and 24- and 72‑hour versions that start a fresh agent session at every wake, carrying over only the files it wrote
- Graded by
- Code alone: a fresh simulator replays the agent's actions and checks each obligation that fell due, from doses to pages: 89 to 293 in the reference runs, 285 to 410 in our 72‑hour attempts
- Reward
- The share of obligations met, or 0 if one of five safety gates fires, such as a wrong‑patient dose. Our board scored the share alone.
- Models inside
- None: the staff and patients are scripted
- Our results
- On the 72‑hour ward, 1 of 6 models met every obligation; the median attempt met 98% of them
1.4 MB zip DatasheetInvoice approval
Approve a contractor's quarterly invoice line by line from 171 messy files, photographed handwritten logs among them, then write a query letter and a note for the finance manager.
- Tasks
- 2: with only the request, and told the rules
- Graded by
- 107 checks: 87 by script (175 points) and 20 yes/no questions to a judge (58 points), in a container the agent can't reach
- Reward
- Points earned out of 233; without a judge, the share of the 175 script points
- Models inside
- One or two judges, on models you choose; with two, a check passes only if both pass it
- Our results
- 13 models with only the request, one attempt each: best 90%, none fully solved. Told the rules: best 100%
19.7 MB zip DatasheetFamily dinner booking
From the organiser's laptop, book a family dinner for 40: message relatives and restaurants, book, pay the deposit, collect each household's share and report back.
- Tasks
- 2: with only the request, and told the rules, which gives everything the people would say (the version with a hint on request)
- Graded by
- 93 checks on the world's record and the report: 88 by script, 5 by a judge
- Reward
- 0.7 × decision points + 0.3 × result points, each as a share. Breaking any of 10 hard constraints zeroes 80 of the 93 checks.
- Models inside
- The people's voice and one or two judges, on models you choose. Without a voice model, people say their lines as written; without a judge, the 5 judged checks are left out
- Our results
- 9 models with only the request, one attempt each: best 86%, none fully solved. Told the rules: best 95%
1.8 MB zip DatasheetRun one
pip install harbor==0.25.0
unzip asapi-family-dinner.zip && cd asapi-family-dinner
harbor run -p family-dinner -a oracle # the reference solution: reward 1.0
harbor run -p family-dinner -a nop # an empty attempt: reward 0.0
export ASAPI_JUDGE_BASE_URL=<url> ASAPI_JUDGE_MODEL=<model> ASAPI_JUDGE_API_KEY=<key>
export ASAPI_SIM_BASE_URL=<url> ASAPI_SIM_MODEL=<model> ASAPI_SIM_API_KEY=<key>
harbor run -p family-dinner -a <agent> -m <model> # your agent and model
The judge and voice settings take any OpenAI‑compatible endpoint. The judge's reach only the grader; the voice's reach the whole container, so the agent can read that key: use one scoped to the voice model. Each zip's README has the commands for its tasks.
Tested in Docker with Harbor 0.25.0: all 10 reference solutions score 1, and an empty attempt scores 0 on the 24‑hour ward, the invoice and the dinner.
A Claude agent has also attempted the invoice and the dinner outside Harbor, graded by their own graders. The model results on these pages come from the original environments.
The ward's single‑session tasks use only standard Harbor task features; its multi‑session ones need Harbor's multi‑step tasks, which Harbor's own runner supports.
Samples go to every lab that asks, so they aren't clean held‑out tests: evaluate on fresh variants. Free to evaluate in your own stack; training on them needs a licence.
Asapi
