The emergency history brief

Build a program that briefs an emergency doctor from a region's raw clinical traffic, following a long contract plus rules that survive only in an old system's filled forms.

Setting: Regional health exchange, emergency care
Deliverable: A working six-stage program, run by hidden tests

Unstated requirementsMessy documentsConflicting sourcesData quality

Models

ran the standard setup

99%

Best score

GPT-6 Astra

98%

Median score

across those models

34 min

Typical run

median time, standard setup

The task

Anamnex, a made-up regional health exchange, wants a program that turns everything the region's hospitals, labs, pharmacies and health plan send into a brief for each emergency arrival: answers to safety questions, a one-page sheet and a filled history form. The agent builds it alone in a Linux container, from a contract of over 4,500 lines, thousands of example files and 22 forms a retired system filled. Nobody answers questions.

The prompt only asks for a program that follows the contract and the old forms. The real test is what a careful engineer of clinical systems would get right: answering only from what had arrived when the patient walked in, never reading silence as "no", linking records only where the sources say they belong together, and recovering the rules nobody wrote down.

What it tests

Recovering rules nobody wrote down

Some of the brief's rules survive only in forms an old system filled, saved as PDF files that must be read without third-party tools.

Reading many messy formats

Hospital message feeds, clinical documents, claims and PDF reports each have their own quirks.

Weighing sources that disagree

Corrections, replacements and conflicting evidence arrive out of order, and the answer depends on what had arrived by a given moment.

Following a long contract exactly

Every output is compared byte for byte or field by field, on an inbox and rule files the agent never saw.

Challenges

1Rules kept only in old forms

Part of the job is described nowhere but in examples of past output, as often happens when a system is replaced.

2As of the moment of arrival

News that arrives after the patient walked in must change nothing, even when it is about something earlier.

3Silence is not "no"

A negative answer needs evidence, and conflicting evidence is an answer of its own.

4People who look alike

Records that share every personal detail can belong to different people, and records merge and split over time.

5Rules files that change

The hidden tests use different codes, units and questions from the examples.

Results

Model	Standard	Time (standard)	Cost (standard)
GPT-6 Astra	99%	42 min	$20.37
GPT-6.1 Sol	98%	34 min	$1.89
GPT-6 Luna	10%	17 min	$0.24

One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.

Setups: Standard, the job as asked.

What models got right and wrong

Three models are on the board, all in Codex CLI. None passed, which needs every one of the 919 hidden tests: GPT-6 Astra passed 909 (98.9%), GPT-6.1 Sol 903 (98.3%) and GPT-6 Luna 90 (9.8%).
GPT-6 Astra and GPT-6.1 Sol each wrote their own reader for the vendor PDF forms with the standard library, matched every field of every shipped form and every shipped reference file, and recovered the rules kept only in the forms; GPT-6.1 Sol passed every test of a single tricky case.
Both lost almost all their tests to one detail in a case no example settles. GPT-6 Astra misread when a medication list proves a patient is not on a kind of drug; GPT-6.1 Sol printed a vaccination date a day early where the same dose was recorded twice. One wrong value reaches the sheet, the form and several other tests.
Both checked the larger example inboxes, which ship no reference briefs, only for valid output, not for what the answers said. GPT-6.1 Sol also accepted one malformed input it should have rejected.
GPT-6 Luna wrote the whole program without running it once and stopped after 17 of its 90 minutes, saying the work was unfinished. One undefined name in its message reader stopped the brief on every inbox, so no brief, sheet or form was ever written.
There is no hinted or told-everything setup for this task: every score is on the job as asked.

Test environment

Kind: Software build from a written contract, for Anamnex, a made-up regional health exchange that wants an emergency-history compiler. The agent works alone in a Linux container; a hidden test suite grades the program
Agent's interface: A terminal in a fresh Docker container (Ubuntu 24.04, Python 3 with pip and pytest), driven by the agent's own command-line tool through harbor: Codex CLI, the only agent on the board (0.133.0 on AWS Bedrock for GPT-6 Astra; 0.160.0 signed in with a ChatGPT plan for GPT-6.1 Sol and GPT-6 Luna)
Documents: 4,490 files in /app: the contract (1 Markdown file, 4,555 lines) in spec/, and in examples/ four example inboxes with their rule files, 149 worked examples, negative cases and 22 history forms an old system filled, saved as PDF files the way vendor software saves them. By format: 2,724 HL7, 536 TXT, 331 XML, 285 X12, 240 JSON, 152 Markdown, 151 PDF, 66 JSONL, 5 LOG
Applications: None provided. The agent writes /app/anamnex itself, in any language; at run time the program may use no network, no clock and no third-party packages, so it must read the PDF forms with the standard library
Web: Not needed: nothing the task asks about is online. The container can reach the internet. Codex CLI, the only agent on the board, ran with its web search turned off (web_search=disabled) for every model, and no run searched the web
People: None: nobody answers questions
Time: None
Actions: Read, write and run files in the container only; nothing outside it changes
Deliverable: One executable, /app/anamnex, with six subcommands (ingest, resolve, extract, reconcile, brief, delta), each run as its own process: they write six JSONL files, a fixed-width text sheet and one filled PDF form per arrival

← PreviousGraduating soonest All 34 tasks Next →The radiation-dose registry