Emergency patient‑history compiler from a long spec
ID tb‑tasks/anamnex‑emergency‑brief · revision 2 · tested 22 September to 22 October 2026
Working alone in a Linux container, the agent builds a six‑stage program for a regional health‑information exchange: for each emergency arrival, it answers 36 safety questions from what had arrived by then, prints a one‑page sheet and fills a PDF history form.
- Checks
- 919, 1 point each, in a hidden test suite
- Tested
- 6 models, one attempt each
- Best
- 100% (Opus 5.5 and Sonnet 5.5)
- Median
- 99%
- Full solves
- 2 of 6
- Median attempt
- 36 min, 100 tool calls
A six‑stage program for emergency arrivals, built from a 4,555‑line contract
Agentic coding
919 hidden tests run the program on unseen inboxes, changed rule sets and malformed inputs, stage by stage.
Multimodal records
Evidence arrives as 4,490 files: message feeds, clinical documents, claims files and PDF reports read by page layout.
Unstated requirements
Five of the nine kinds of answer are documented nowhere and survive only in 22 forms the retired system filled.
4,490 files, a long contract and a hidden test suite
The agent's world
A shell in a fresh container with Python and pytest; the internet is on while it works, but the program may not use it.
Hidden inputs
The tests run the program on a held‑out inbox of about 4,400 files, under two changed rule sets, and on 53 malformed inputs.
Limits
90 minutes for the agent and 30 for the tests, with nobody to ask; an expert would need about 12 hours.
The reward
The share of 919 hidden tests passed; the task's own pass needs all 919, and one wrong value fails about ten.
Two programs passed all 919 tests, three came within ten, and one passed only 90
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 100% |
| Sonnet 5.5Anthropic | 100% |
| Fable 5.1Anthropic | 99% |
| GPT‑6.1 SolOpenAI | 99% |
| GPT‑6 AstraOpenAI | 99% |
| GPT‑6 LunaOpenAI | 10% |
They stop once the examples match, accept a malformed input, and never run the program
Agentic coding
One matched every shipped reference, then checked the larger example inboxes only for well‑formed output, and lost 10 tests where no shipped answer exists.
Two accepted a plain file where an inbox folder is required, their only failed test.
One wrote its program without running any stage, and one undefined name stopped every inbox: 829 of 919 tests failed.
What an attempt costs, and what the environment includes
Cost
Estimated from tokens, attempts cost about $0.24 to $23.62 for the agent model; grading is a script, with no model calls.
What's in it
The contract, worked examples, example inboxes and forms, the hidden test suite with its fixtures, and the reference solution.
Format
Packaged as a Harbor task on request; it runs in Docker, and grading is the hidden test suite alone.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials