Radiation‑dose registry program from a long spec
ID tb‑tasks/grayledger‑dose‑registry · revision 6 · tested 18 September to 22 October 2026
Working alone in a Linux container, the agent builds a six‑stage program from a 2,448‑line contract: it reads a medical‑imaging archive, keeps each patient's dose record, writes two fixed‑width filings and judges whether a proposed procedure stays within the limits.
- Checks
- 466, 1 point each, in a hidden test suite
- Tested
- 12 models, one attempt each
- Best
- 100% (8 models)
- Median
- 100%
- Full solves
- 8 of 12
- Median attempt
- 14 min
A six‑stage dose registry, built alone from a 2,448‑line contract
Agentic coding
466 hidden tests run the program on a held‑out archive and a changed rules folder, comparing its filings byte for byte.
A binary imaging archive, a long contract and a hidden test suite
The agent's world
A shell in a fresh container with Python and pytest, no imaging library, and the contract's own tables for every field.
Hidden inputs
After the agent stops, the tests run the program on a held‑out archive, the same under changed rules, and 12 malformed inputs.
Limits
90 minutes for the agent and 15 for the tests, with nobody to ask; an expert would need about 10 hours.
The reward
The share of 466 hidden tests passed; the task's own pass needs all 466, and every test runs the program itself.
67% of attempts passed all 466 tests; two lost a few, and two had no working program
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| DeepSeek V4.1 FlashDeepSeek | 100% |
| Fable 5.1Anthropic | 100% |
| GPT‑5.6 SolOpenAI | 100% |
| GPT‑6 AstraOpenAI | 100% |
| GPT‑6.1 SolOpenAI | 100% |
| Opus 5Anthropic | 100% |
| Opus 5.5Anthropic | 100% |
| Qwen3.8‑27BAlibaba | 100% |
| Sonnet 5.5Anthropic | 99% |
| Opus 4.8Anthropic | 98% |
| GPT‑6 LunaOpenAI | 4% |
| GLM‑5.3‑FlashZhipu | 0% |
They check only against the answers they had, skip the changed rules, and never run the program
Agentic coding
One matched every shipped answer, reported its work verified, and lost 10 tests to one rule: which of two same‑day records counts.
One never ran its program under a changed rules folder and lost 6 tests there.
One only checked that its code parsed, as its last message said, and passed 18 of 466 tests.
One read the contract for 70 minutes in 10 model calls, reasoning at ever greater length, and saved nothing.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $2.80 at list prices for the agent model; grading is the test suite alone.
What's in it
The contract, worked examples and example archives, the hidden test suite with its held‑out archive and changed rules, and the reference solution.
Format
Packaged as a Harbor task on request; it runs in Docker, and grading is the hidden test suite alone.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials