AsapiThe AI Agent Trials
Asapi / Catalogue / Emergency patient‑history compiler from a long spec
Environment datasheet · Health records software

Emergency patient‑history compiler from a long spec

ID tb‑tasks/anamnex‑emergency‑brief · revision 2 · tested 22 September to 22 October 2026

Working alone in a Linux container, the agent builds a six‑stage program for a regional health‑information exchange: for each emergency arrival, it answers 36 safety questions from what had arrived by then, prints a one‑page sheet and fills a PDF history form.

Checks
919, 1 point each, in a hidden test suite
Tested
6 models, one attempt each
Best
100% (Opus 5.5 and Sonnet 5.5)
Median
99%
Full solves
2 of 6
Median attempt
36 min, 100 tool calls
What it tests

A six‑stage program for emergency arrivals, built from a 4,555‑line contract

Agentic coding

919 hidden tests run the program on unseen inboxes, changed rule sets and malformed inputs, stage by stage.

Multimodal records

Evidence arrives as 4,490 files: message feeds, clinical documents, claims files and PDF reports read by page layout.

Unstated requirements

Five of the nine kinds of answer are documented nowhere and survive only in 22 forms the retired system filled.

How it works

4,490 files, a long contract and a hidden test suite

The agent's world

A shell in a fresh container with Python and pytest; the internet is on while it works, but the program may not use it.

Hidden inputs

The tests run the program on a held‑out inbox of about 4,400 files, under two changed rule sets, and on 53 malformed inputs.

Limits

90 minutes for the agent and 30 for the tests, with nobody to ask; an expert would need about 12 hours.

The reward

The share of 919 hidden tests passed; the task's own pass needs all 919, and one wrong value fails about ten.

Results

Two programs passed all 919 tests, three came within ten, and one passed only 90

Each dot is one attempt, and the line marks the median.

MedianFully solved
6 modelsOne attempt each99%2 of 6
ModelReward
Opus 5.5Anthropic100%
Sonnet 5.5Anthropic100%
Fable 5.1Anthropic99%
GPT‑6.1 SolOpenAI99%
GPT‑6 AstraOpenAI99%
GPT‑6 LunaOpenAI10%
What models get wrong

They stop once the examples match, accept a malformed input, and never run the program

Agentic coding

  1. One matched every shipped reference, then checked the larger example inboxes only for well‑formed output, and lost 10 tests where no shipped answer exists.

  2. Two accepted a plain file where an inbox folder is required, their only failed test.

  3. One wrote its program without running any stage, and one undefined name stopped every inbox: 829 of 919 tests failed.

Running it

What an attempt costs, and what the environment includes

Cost

Estimated from tokens, attempts cost about $0.24 to $23.62 for the agent model; grading is a script, with no model calls.

What's in it

The contract, worked examples, example inboxes and forms, the hidden test suite with its fixtures, and the reference solution.

Format

Packaged as a Harbor task on request; it runs in Docker, and grading is the hidden test suite alone.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.