Radiation‑dose registry program from a long spec

A complete contract, graded partly on inputs the agent never sees

Role

The engineer for a made‑up company that sells a radiation‑dose registry to hospital networks, building its engine alone. Nobody answers questions.

Request

Build the program the contract specifies, after reading it and its worked examples: six subcommands, each a separate process. It warns that an undocumented dose is a range, not zero, that the limits depend on the patient on the date asked, and that grading uses unseen inputs.

Deliverable

One executable with six stages (index, extract, reconcile, submit, amend, assess), each a separate process that builds on earlier stages' files: a study index, dose events, a per‑patient ledger, a fixed‑width filing with its flags, a fixed‑width amendment and a verdict per proposed procedure.

What the reward depends on

Decoding the binary archive with the contract's own dictionaries; counting each exposure once, from the record that counts; bounding missing doses as ranges; reading the patient's details as of the date asked; taking every rule from the rules folder; byte‑exact filings; a yes, no or cannot‑tell verdict.

Where the requirements are

All in the contract and its 35 worked examples, and most rules have an obvious reading the contract rules out. Of the three example archives, only one ships the outputs a correct program writes.

A bare container, a long contract and three example archives

Interface

A shell in a fresh Docker container (Ubuntu, Python 3 with pip and pytest), driven by each model's own agent harness (Claude Code, Codex CLI or hermes in our attempts). The internet is on for installing packages; nothing the task needs is online.

Files

89: the contract (2,448 lines), a file of 35 worked examples, and three example archives with 52 binary medical images, 18 rule files, patient‑system exports and query lists, and one archive's reference outputs.

Libraries

No imaging library is supplied, and the request rules one out for definitions: fields come from the contract's own tables. The agent may install other tools.

People and time

None: no simulated people, services or clock. The agent gets the request once and works until it stops or reaches its 90‑minute limit.

Hidden inputs

After the agent stops, the tests run its program on the example archives, a held‑out archive, the same archive under a changed rules folder, and 12 malformed inputs. Tests and reference outputs are copied in only then.

Stage isolation

Each stage runs as a separate process. A decoy folder is planted after indexing, and the stages after extraction run with the archive moved away, so a later stage that rereads it fails.

Budget

90 minutes for the agent and 15 for the tests; up to 500 turns for hermes; no spend cap. The task's metadata puts an expert at 10 hours.

466 hidden tests, one point each, all decided by code

Structure

466 tests from 66 test functions: 56 compare figures with references or with each other, 302 check that a rule was applied, 82 check the inferred dose ranges and verdicts, and 26 check form: stages that run alone, repeatable output, silence and exit codes.

Tricky cases

257 of the rule tests: 256 cases, one test each, plus one test that the list isn't empty. The cases fall into 75 groups, each citing the contract section that states its rule.

Changed rules

The held‑out archive is also graded under a changed rules folder, with other units, codes and limits, so constants copied from the examples fail.

No empty passes

Every test runs the program as a subprocess, so none passes without it, and the property tests check their own counts, so an empty output can't pass them.

Who decides

Code alone: the hidden pytest suite. No LLM judge and no written rulings.

Pass and reward

The verifier's reward is 1 only if all 466 pass, as the request warns. Our board scored the share passed.

Steep losses

The suite runs the program on its five archives in one shared step that stops at the first crash, and 440 tests read from it. In the audit's 12 edited copies of the reference solution, each with one rule changed, the loss was 2 to 24 tests.

Formula

Reward = tests passed / 466, from 0 to 1; the task's own pass is 466 of 466.

Eight of twelve attempts passed all 466 tests

8 of the 12 attempts, one per model, with only the request, passed every test on inputs they had never seen, two of them by open‑weight models. Of the other four, two lost 10 and 6 tests, one never ran its program and one wrote none.

Each dot is one model's attempt; the line marks the median.

One attempt per model, with only the request. There is no version with a hint or told the rules.

An attempt that wrote no program counts as 0. One attempt reached the 90‑minute limit with its program already complete, and was graded as it stood.

Four attempts used the package frozen on 18 September, which wasn't kept; the other eight used the rebuild of 21 September. Both have the same request and the same 466 tests.

What the models got wrong

Counted over the 12 models, one attempt each, with only the request.

1. Checked only against the answers it had. 1 of 12 attempts matched every shipped answer, reported its work verified, and lost 10 tests to one rule, which of two same‑day records counts. 9 of 12 got it right.

2. Never run under the changed rules. 1 of 12 read every report‑level control as the declared event count, where the contract names only one, never ran its program under a changed rules folder, and lost 6 tests there.

3. Never ran the program. 1 of 12 wrote its program and only checked that the code parsed, as its last message said: 18 of 466 tests passed.

4. No program at all. 1 of 12 read the contract and examples for 70 minutes in 10 model calls, reasoning at ever greater length, and saved nothing.

What an attempt costs

Length

With only the request, the median attempt took 14 minutes, from 6 to 90 across the 12.

Cost

Recorded for 10 of the 12 attempts: median $2.82, from $0.06 to $14.44, $43.33 in all, for the agent model only. Six of the ten are estimates: five for attempts on a plan not billed per call, and one harness's upper‑bound estimate.

What you license, and what's public

Format

Packaged as a Harbor task on request. It isn't one of our three downloadable samples.

With a licence

The contract, worked examples and example archives, the hidden test suite with its held‑out archive, changed rules folder and malformed inputs, the reference solution, and the grader's report on each of the 11 attempts that wrote a program.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

Files, tests and reference solution aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model on our board, each in its own harness (Claude Code, Codex CLI or hermes), so model and harness effects are mixed.
• Outside our board, the task's notes record 20 attempts at this build by the first nine models tested, ours among them: 8 full solves, and each model with two or more failed at least once. The other 11 aren't in the repository, so we couldn't check them.
• The share pays for partial programs. By an audit's figures, a program that leaves out the verdicts scores 64%; the reference solution with one crash, on the changed‑rules archive alone, scores 6%.
• The contract promises a worked example for every tie‑break, but the same‑day tie behind Opus 4.8's near miss has none; an example archive holds one, without its answer. An audit judged the rule unambiguous.
• The tests read the example archives from the agent's own folder, which it can change. An audit found that this lets no wrong program pass, and that the held‑out archive's identifiers appear nowhere the agent can see.
• When an audit ran the hidden suite offline, the reference solution passed all 466 tests and an empty attempt none; the six programs then on our board matched their stored results test for test.
• The task's notes say the difficulty is concentrated: in the authors' attempts, the held‑out archive under the changed rules folder decided almost every failure. On our board, Sonnet 5.5's six losses were all there, while Opus 4.8's near miss lost tests on three archives.
• No red‑team attempts yet.