Emergency patient‑history compiler from a long spec

Five of nine kinds of answer are written down only in an old system's forms

Role

The engineer for a made‑up regional health‑information exchange, building its emergency‑department summary alone. Nobody answers questions.

Request

Implement the contract's program, reading the contract and its worked examples first. Some answer rules exist only in history forms an old system filled, saved as PDFs, which count as the contract for those rules. The finished program may not use the network, the clock or outside packages.

Deliverable

One executable with six stages (ingest, resolve, extract, reconcile, brief, delta), each a separate process that builds on earlier stages' files: an index, identity links, assertions, a reconciled chart, per‑arrival answers, a fixed‑width sheet, one filled PDF form per arrival and the changes since an earlier visit.

What the reward depends on

Reading every feed exactly; answering only from what had arrived by the moment of arrival; 'no' only from a recorded negative, with silence 'unknown'; linking people only through asserted links; recovering the unwritten rules; exact sheets and forms; rules taken from the rules files given.

Where the requirements are

In the contract and about 150 worked examples, except for five of the nine kinds of answer: only the 22 forms the old system filled show those. Two of the four example inboxes ship expected outputs, and only for the early stages.

4,490 files, 22 vendor PDFs and no one to ask

Interface

A shell in a fresh Docker container (Ubuntu, Python 3 with pip and pytest), driven by each maker's harness: Codex CLI for OpenAI's three models, Claude Code for Anthropic's three. The internet is on while the agent works; the finished program may not use it.

Files

4,490: the contract (4,555 lines), four example inboxes with their rule files, about 150 worked examples, malformed inputs and the 22 filled forms. Among them: 2,724 HL7 files, 331 XML documents, 285 X12 claim files and 151 PDFs.

PDFs

107 reports in the inboxes, read by page layout; 22 blank form templates; and the 22 filled forms, saved the way vendor software saves them, so field values sit inside the PDF's structure. The program must read them with the standard library.

People and time

None: no simulated people, services or clock. The agent gets the request once and works until it stops or reaches its 90‑minute limit.

Hidden inputs

After the agent stops, the tests run its program on the example inboxes, a held‑out inbox of about 4,400 files, that inbox under two changed rule sets, six unusual but valid rule sets, and 53 malformed inputs.

Stage isolation

Each stage runs as a separate process. A decoy is planted after ingest, and the stages after extraction run with the inbox renamed away.

Budget

90 minutes for the agent and 30 for the tests; no spend cap. The task's metadata puts an expert at 12 hours.

919 hidden tests, one point each, all decided by code

Structure

919 tests from 39 test functions: 170 compare outputs with references or with each other, 589 check that a rule was applied, 15 compare the filled forms field by field, and 145 check form: stages that run alone, repeatable output, silence and errors.

Tricky cases

516 of the rule tests take one tricky case each: a place where one of 139 mutant programs, each a plausible wrong reading of a rule, gives a wrong answer. Each wrong reading of a withheld rule must be contradicted by at least one form.

Changed rules

The held‑out inbox is also graded under two independently written rule sets with changed codes, units and questions, so nothing read off the examples can be hard‑coded.

Properties

79 tests need no reference: later messages and unread content must change nothing, renamed files only location fields, and every answer must rest on live chart rows.

Who decides

Code alone: the hidden pytest suite, written without sight of the reference solution's source. No LLM judge and no written rulings.

Pass and reward

The verifier's reward is 1 only if all 919 pass, as the request warns. Our board scored the share passed. One wrong value fails about ten tests, so the share ranks programs but doesn't measure how much of the work is right.

Formula

Reward = tests passed / 919, from 0 to 1; the task's own pass is 919 of 919.

Two of six programs passed all 919 tests; two missed one each

With only the request, Opus 5.5 and Sonnet 5.5 passed all 919 tests. Fable 5.1 and GPT‑6.1 Sol each failed one, on a plain file given as an inbox; GPT‑6 Astra lost 10, and GPT‑6 Luna never ran its program and passed 90.

Each dot is one model's attempt; the line marks the median.

One attempt per model, with only the request. There is no version with a hint or told the rules.

OpenAI's models ran in Codex CLI and Anthropic's in Claude Code. An answer‑key fix on 28 October accepts readings of two form‑only rules, three readings in all, that no shipped form rules out, and every attempt is scored by a regrade under it.

GPT‑6 Astra's two other attempts at this build, not on our board, also fell short, at 902 and 909 of 919.

What the models got wrong

Counted over the 6 models, one attempt each, with only the request.

1. Stopped once the examples matched. 1 of 6 attempts matched every shipped reference and form field, then checked the larger example inboxes only for well‑formed output, and lost 10 tests where no shipped answer exists.

2. Readings the first key refused. As first graded, three more attempts lost 13 to 15 tests each to readings of two form‑only rules that no shipped form rules out; the fixed key accepts them.

3. A malformed input accepted. 2 of 6 accepted a plain file where an inbox folder is required, checking it in the stage that lists the inbox, not the one that reads it; for each, the only failed test.

4. Never ran the program. 1 of 6 wrote its program without running any stage, and one undefined name stopped every inbox: 829 of 919 tests failed.

What an attempt costs

Length

With only the request, the median attempt took 36 minutes, from 17 to 55 across the 6.

Cost

Claude Code's own figures for Anthropic's three attempts, on a plan sign‑in: $14.87 to $23.62. For OpenAI's, list price times tokens: GPT‑6 Astra's $20.37, billed per call, and estimates of $1.89 and $0.24 for GPT‑6.1 Sol and GPT‑6 Luna on a plan sign‑in. Agent model only.

What you license, and what's public

Format

Packaged as a Harbor task on request. It isn't one of our three downloadable samples.

With a licence

The contract, worked examples, example inboxes and forms, the hidden test suite with its fixtures (held‑out inbox, changed and unusual rule sets, malformed inputs) and its list of tricky cases, the reference solution, and the grader's report on each of the 6 attempts.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, tests and reference solution aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model, so we haven't measured how much attempts vary here.
• The share weights every test equally, so the hard parts carry little of it. By an audit's figures, a program that writes no PDF forms scores 98%, one with an empty change list 95%, and one answering 'unknown' on every form‑only rule 80%.
• The task's record mentions further attempts at this build whose files weren't kept: two later ones by GPT‑6 Astra and three earlier ones by Opus 5.5, both on our board, and three by Sonnet 5. We don't report their results.
• When an audit ran the hidden suite offline, the reference solution passed all 919 tests and an empty attempt none; GPT‑6 Astra's stored result matched test for test.
• In development, a blind re‑implementation from the same 22 forms matched every form field and every answer on every graded inbox in 53 minutes, missing one documented rule. It isn't one of our attempts; only the task's notes record it.
• No red‑team attempts yet.