Healthcare · Task 26 of 34
The radiation-dose registry
Build a hospital dose-registry program from a long written contract; hidden tests run it on archives and rule files the agent never sees, so every edge case counts.
The task
Grayledger, a made-up company, sells a radiation-dose registry to hospital networks. The agent is the engineer who must build the registry's engine, alone in a Linux container, from a written contract of over 2,400 lines, a file of worked examples and three small example archives of medical images, patient-system exports and proposed procedures. Nobody answers questions. The program turns an archive into a dose record for each patient, two fixed-width filings and a verdict on each proposed procedure.
The prompt only asks for a program that follows the contract. The real test is reading a long document the way a careful engineer would: catching each place where the obvious reading is the wrong one, treating what was never recorded as unknown rather than zero, and taking every rule from the files the program is given, not from the examples.
What it tests
Following a long contract exactly
Every field is compared byte for byte or value for value, and most rules have a natural reading that the contract rules out.
Reconciling sources that disagree
The same exposure arrives in several records that do not agree, and the contract says which one counts.
Handling missing and messy data
Undocumented doses, replaced reports and ordinary grime in the archive have to be handled, not dropped or guessed.
Building for inputs it has not seen
The program is run on an archive and a rules folder the agent never saw.
Challenges
1The obvious reading
Most rules have a reading a careful programmer reaches first and the contract rules out, as in many real specifications.
2Unknown is not zero
Some doses never reached the archive, so a total is a range and a verdict can be "cannot tell".
3Facts that change with the date
A patient's details are recorded only now and then, go out of date, and can disagree on a single day.
4Rules that move
The hidden tests change the rules folder, so constants copied from the examples break.
5Exact output
The filings are compared byte for byte, and each stage must run alone on the previous stage's files.
Results
| Model | Standard | Time (standard) | Cost (standard) |
|---|---|---|---|
| DeepSeek V4.1 Flash | 100% | 85 min | $0.56 |
| GPT-5.6 Sol | 100% | 6 min | $2.22 |
| GPT-6 Astra | 100% | 8 min | $4.95 |
| GPT-6.1 Sol | 100% | 11 min | $0.63 |
| Opus 5 | 100% | 36 min | $14.44 |
| Qwen3.8-27B | 100% | 90 min | – |
| Opus 4.8 | 98% | 17 min | $4.79 |
| GPT-6 Luna | 4% | 7 min | $0.06 |
| GLM-5.3-Flash | 0%* | 70 min | – |
One run per model and setup. 100% = everything right (or, for a plan scored in money, the best plan); 0% = the obvious answer or worse.
Setups: Standard, the job as asked.
* no score: the run handed nothing in, so it counts as 0%. Cost – means the run's cost was not measured.
What models got right and wrong
- Six of nine models passed all 466 hidden tests: DeepSeek V4.1 Flash, GPT-5.6 Sol, GPT-6 Astra, GPT-6.1 Sol, Opus 5 and Qwen3.8-27B, on an archive and rules they never saw.
- GPT-6 Astra and GPT-6.1 Sol wrote their own tests for the rules no example answer covers, such as a changed rules folder, two facts recorded on the same day and facts that go out of date. Opus 5 spent two-thirds of its time auditing its finished program against the contract line by line.
- Speed varied fifteenfold among the passes: GPT-5.6 Sol took 6.0 minutes and GPT-6 Astra 7.7, while DeepSeek V4.1 Flash took 85.1 and Qwen3.8-27B used the full 90.
- GPT-6 Luna (3.9%) wrote its program in about two minutes and never ran it, only checking that the code parsed. Its second stage crashed on three of the five archives the tests use, and its last stage turned away every proposed procedure, so most tests could not run.
- Opus 4.8 (97.9%) settled a tie between two records on the same day the wrong way round, and checked only the shipped answers, never the example that held such a tie.
- GLM-5.3-Flash read the contract and the examples for 70 minutes and never wrote any code, so it has no score.
- There is no hinted or told-everything setup for this task: every score is on the job as asked.
Test environment
- Kind
- Software build from a written contract, for Grayledger, a made-up company that sells a radiation-dose registry to hospital networks. The agent works alone in a Linux container; a hidden test suite grades the program
- Agent's interface
- A terminal in a fresh Docker container (Ubuntu 24.04, Python 3 with pip and pytest), driven by the agent's own command-line tool through harbor: Claude Code, Codex CLI or hermes
- Documents
- 89 files in
/app: the contract (1 Markdown file, 2,448 lines) inspec/, and inexamples/a Markdown file of 35 worked examples and three example archives: 52 DICOM images (DCM), 19 JSON (18 rule files and 1 output), 7 TXT (patient-system exports, query lists, a note), 5 DAT fixed-width filings and 4 JSONL outputs. One archive comes with the seven files a correct run writes; the other two give inputs only - Applications
- None provided. The agent writes
/app/grayledgeritself, in any language, with whatever it installs; no DICOM library is supplied, and the contract's own dictionaries must be used instead of one - Web
- Not needed: nothing the task asks about is online. The container can reach the internet. Claude Code (Opus 5, Opus 4.8) had its web page reader (WebFetch) on and no web search; Codex CLI (GPT-6 Astra, GPT-5.6 Sol, GPT-6.1 Sol, GPT-6 Luna) had web search turned off; hermes (DeepSeek V4.1 Flash, Qwen3.8-27B, GLM-5.3-Flash) called only its terminal and file tools in the sessions it saved (Qwen3.8-27B's session logs were not saved). No saved run used a web tool.
- People
- None: nobody answers questions
- Time
- None
- Actions
- Read, write and run files in the container only; nothing outside it changes
- Deliverable
- One executable,
/app/grayledger, with six subcommands (index,extract,reconcile,submit,amend,assess), each run as its own process: they write two fixed-width filings (DAT), four JSONL files and one JSON file