AsapiThe AI Agent Trials
Asapi / Catalogue / Radiation‑dose registry program from a long spec
Environment datasheet · Hospital software

Radiation‑dose registry program from a long spec

ID tb‑tasks/grayledger‑dose‑registry · revision 6 · tested 18 September to 22 October 2026

Working alone in a Linux container, the agent builds a six‑stage program from a 2,448‑line contract: it reads a medical‑imaging archive, keeps each patient's dose record, writes two fixed‑width filings and judges whether a proposed procedure stays within the limits.

Checks
466, 1 point each, in a hidden test suite
Tested
12 models, one attempt each
Best
100% (8 models)
Median
100%
Full solves
8 of 12
Median attempt
14 min
What it tests

A six‑stage dose registry, built alone from a 2,448‑line contract

Agentic coding

466 hidden tests run the program on a held‑out archive and a changed rules folder, comparing its filings byte for byte.

How it works

A binary imaging archive, a long contract and a hidden test suite

The agent's world

A shell in a fresh container with Python and pytest, no imaging library, and the contract's own tables for every field.

Hidden inputs

After the agent stops, the tests run the program on a held‑out archive, the same under changed rules, and 12 malformed inputs.

Limits

90 minutes for the agent and 15 for the tests, with nobody to ask; an expert would need about 10 hours.

The reward

The share of 466 hidden tests passed; the task's own pass needs all 466, and every test runs the program itself.

Results

67% of attempts passed all 466 tests; two lost a few, and two had no working program

Each dot is one attempt, and the line marks the median.

MedianFully solved
12 modelsOne attempt each100%8 of 12
ModelReward
DeepSeek V4.1 FlashDeepSeek100%
Fable 5.1Anthropic100%
GPT‑5.6 SolOpenAI100%
GPT‑6 AstraOpenAI100%
GPT‑6.1 SolOpenAI100%
Opus 5Anthropic100%
Opus 5.5Anthropic100%
Qwen3.8‑27BAlibaba100%
Sonnet 5.5Anthropic99%
Opus 4.8Anthropic98%
GPT‑6 LunaOpenAI4%
GLM‑5.3‑FlashZhipu0%
What models get wrong

They check only against the answers they had, skip the changed rules, and never run the program

Agentic coding

  1. One matched every shipped answer, reported its work verified, and lost 10 tests to one rule: which of two same‑day records counts.

  2. One never ran its program under a changed rules folder and lost 6 tests there.

  3. One only checked that its code parsed, as its last message said, and passed 18 of 466 tests.

  4. One read the contract for 70 minutes in 10 model calls, reasoning at ever greater length, and saved nothing.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $2.80 at list prices for the agent model; grading is the test suite alone.

What's in it

The contract, worked examples and example archives, the hidden test suite with its held‑out archive and changed rules, and the reference solution.

Format

Packaged as a Harbor task on request; it runs in Docker, and grading is the hidden test suite alone.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.