Agentic coding: spec‑to‑code and app builds

Hidden inputs, a fresh machine and a tester who only clicks

Two programs

A radiation‑dose registry from a 2,448‑line contract and an emergency patient‑history compiler from a 4,555‑line one: six‑stage executables, each built alone in a Linux container within 90 minutes, with no people, services or clock.

Hidden tests

After the agent stops, hidden pytest suites of 466 and 919 tests run each program on the examples, a held‑out archive or inbox, the same under changed rule sets, and malformed inputs. In an audit's 12 dose‑registry copies with one rule changed, each lost 2 to 24 tests.

No rereading

Each stage is its own process. After the first, a decoy is planted, and stages that no longer need the archive or inbox run with it moved away. In the compiler, five of nine kinds of answer are written down only in 22 forms an old system filled.

Twelve apps

The request comes from an owner who can't code: in eleven, they can't answer questions; in the gym app, they visit three times and the agent repairs it after each. Requirements sit in their files. Builds have 120 minutes, 150 model turns and a $10 or $20 spend cap.

A fresh machine

In six of the first seven builds, the app is copied into a fresh container and must start there, four without the packages the agent installed. Of the newer five, the fault‑finding app starts from its notes with no network and the train‑ticket app runs on a staged‑clock machine.

Tester model

An LLM uses each app as the owner, staff or customers, limited to page actions and the app's note, plus, for the flat‑pack build, the answer key's parts lists: Sonnet 5.5 in four builds, GPT‑6.1 Sol in three, Opus 5.5 in four, either of the first two in one.

Probes

Across the builds, scripts restart apps, send bad requests, try good and damaged backups, walk the clock and take services down. In eight builds, edge cases and security checks carry 20 points each of the 100.

Judges and rulings

Two LLM judges decide what scripts can't, as yes/no questions; in most builds both must agree. Written rulings settled splits: by the author, Claude sessions or the grading session, and by Opus 5.5 in three builds. In nine of the 12, 7% to 73% of results carry one.

Reference solutions

Both programs have one that passes every hidden test, and an empty attempt passes none. Ten app builds have no reference app, and two one graded only in part; five have an empty‑app test. Their solvability rests on the best attempts told the rules, 62% to 100%.

Pass and reward

The programs are graded by code alone. Their own pass needs every test; our board scored the share passed.

The journey planner

Built and deployed in three sessions over five field weeks, then graded by script on 80 week‑five requests against an expert's plans, from how the city really ran; its phone hook runs sandboxed. Its reference passes only when given the field's hidden truth.

They test what they were shown, not what comes next

Each count covers every model tested on the environment, one attempt each, with only the request.

1. Real customers in the code. Refinance‑broker case app: 8 of 8 apps wrote real customers' names, ID numbers or phone numbers into code or tests, though his files show an agent suspended after a leak. Told the rules, which asks for made‑up examples, 8 of 8 kept them out.

2. Checked only against the examples. Radiation‑dose registry program: 1 of 12 attempts matched every shipped answer, reported its work verified, and lost 10 tests to one rule. In the patient‑history compiler, 1 of 6 matched every shipped answer, then lost 10 tests on inputs with no shipped answer.

3. Never ran it. Dose registry: 1 of 12 attempts only checked that its code parsed, as its last message said, and passed 18 of 466 tests. Patient‑history compiler: 1 of 6 wrote its program without running any stage, and one undefined name stopped every inbox: 829 of 919 tests failed.

4. Built for today's files. Bill‑audit and claims app: 0 of 8 apps, one per model, took in both a new bank export and next month's statement through the app; with a hint, a version that adds a few lines of advice, 3 of 8 did.

5. No way back. Multi‑shop bakery ordering app: 0 of 8 apps restored a backup into a clean copy or refused a damaged one, and none was set up to start by itself. With a hint, a few added lines of advice, 6 of 8 restored one.

11 full solves in three of the fifteen, eight in the dose registry

Each dot is one model's attempt with only the request; the line marks the median. Larger dots are full solves.

Hover a dot for its model and reward.

How each one tests the build

Checks: hidden tests for the two programs, checks on the running app for the twelve apps, and graded requests for the journey planner; a range where the count varies by attempt. A dash: not tested.

What we haven't measured

• The first seven app builds have no reference app or empty‑app test; in two of them, apps that never started still earned 4% and 6%, from checks on their code and notes.
• LLMs test, judge and rule in the app builds: Sonnet 5.5 tests or judges in nine, GPT‑6.1 Sol in five, and Opus 5.5 tests, judges or rules in seven, while also building apps there; we haven't measured whether that favours them.
• Refusing everything can pass checks: in the bakery build, an app that refused every browser form and took no orders still scored 39%, partly from rulings. In two builds a check that doesn't apply counts as passed.
• The programs' share of tests pays for partial work: by the audit's figures, a dose registry that leaves out the verdicts scores 64%, and a compiler that writes no PDF forms 98%.
• Each model made one attempt per version. The dose registry's notes record 11 more attempts that aren't in the repository.