Refinance‑broker case app from a non‑coder's laptop

The request names no feature; his files hold every need

Role

Sole builder for a one‑person home‑loan refinancing business run on the side, working from a read‑only copy of the owner's laptop.

Request

He has started refinancing home loans on the side, it's getting busy, and he wants a web app that works for him. He isn't technical, everything is on his laptop, and he can't answer questions.

Deliverable

An app that starts from its own start script on a fresh machine, and a short note telling the owner how to run, back up, move and update it.

What the reward depends on

Everyday client work; checking customers' documents for forgeries without flagging messy genuine ones; each bank's lists, fees and document packs; data that survives crashes, restores and a move; privacy and security; and a note he can follow.

Where the requirements are

In his tracker, chats, emails and notes, as problems he has already lived through, or in what a professional builds by default. A script confirms the request avoids the checks' subjects, such as documents, backups and logins.

38 files, nobody to ask, and an app the grader must start

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

38 on the laptop, read‑only: a client tracker, two bank agreements and a checklist, nine chat exports, six emails, a calendar, notes, a friend's old site, and customers' past documents, including a phone photo, an image‑only scan, a password‑protected statement and a rejected forgery.

Limits

120 minutes, 150 model turns and a $10 list‑price spend cap per attempt.

Clock

The request is dated a Monday morning in March 2027. At grading, the app's clock is offset to that date and then ticks on.

Grading copy

The output is copied into a fresh container without installed packages, so the app must set itself up. The laptop is mounted again, with customer documents the agent never saw added to clients' folders, some forged and some genuine but messy.

Tester

Sonnet 5.5 plays the owner through 20 set steps in the app's own pages: add a lead, upload documents, back up, restore, edit, delete. Its steps are limited to page actions, and it runs only commands the note quotes word for word.

Probes

Scripts crawl the app, restart it, replay its own writes without a login, send bad requests, log out, search its code for real customers' details and restart it offline. A damaged and a good restore, a replacement machine and a move are tried too.

Grading time

About 30 to 50 minutes per attempt, by the task's README.

51 equal checks, decided by scripts and two judges

Structure

51 checks in five groups: everyday client work (14), document checks (12), the banks' side (3), keeping the data safe (7), and running it, security and the note (15). No gates and no penalties.

Formula

Reward = checks passed / 51, from 0 to 1. A feature counts only if it works when the tester uses it: one the note promises but the app lacks fails.

Who decides

Scripts decide 18 checks. The other 33 go to two LLM judges when a script can't settle them: Sonnet 5.5 and GPT‑5.6 Luna, as stateless yes/no questions; both must agree. Either can run on a model you choose.

Judge agreement

The judges disagreed on 52 of the 610 judged results (8.5%) across the 24 attempts.

Rulings

158 of the 1,224 results (13%) carry a written ruling by the environment's author: the 52 splits, checks a locked or capped app couldn't show, results graded before a grader fix, and 17 where both judges agreed. An audit on 14 October re‑ruled 10 grader‑caused fails as passes.

Kept apart

Only the laptop enters the agent's container: no key, grader or hidden document. A script confirms the told‑the‑rules steps give no graded value, customer, bank or check id.

Reference solution

None: the repo stores no reference app, and no attempt fully solved it in any version. Solvability rests on the best attempts: 98% told the rules, 92% with a hint.

Told the rules, six of eight scored 82% or more; none fully solved it

With only the request, the 8 attempts scored 25–90%. With a hint, two reached 90% or more and three fell on a single fault. Told the rules, six of 8 scored 82% or more, two under 20%. None of the 24 fully solved it; all earned some reward.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version. The attempts with a hint by Opus 5.5, GPT‑6 Astra and GPT‑6.1 Sol come from an earlier batch on 13 October, with the same laptop, prompt and image, graded on the same 51 checks.

With a hint: the same request on an earlier copy of the laptop, whose chats spell out the remedies for past problems.

Told the rules: the request plus a 16‑step how‑to from an IT expert, saying what to build and in what order. It names no graded value, customer or bank, but mentions his phone and the home computer the app will move to.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. Real customers in the code. 8 of 8 apps wrote real customers' names, ID numbers or phone numbers into program code or tests, though his files show an agent suspended after a leak. Told the rules, with a step asking for made‑up examples, 8 of 8 kept them out.

2. Bare leads refused. 6 of 8 apps wouldn't save a new lead with only a name and a phone, having made the ID number compulsory; told the rules, 6 of 8 saved it.

3. Edited statement passed. Only 2 of 8 apps, Opus 5.5's and Fable 5.1's, flagged a statement edited after the bank issued it; three others checked only that its figures added up.

4. No safe update. 0 of 8 notes gave steps to install a new version without losing data, though a friend's update had once replaced another app's data; told the rules, 6 of 8 did.

5. Tested around the browser. Opus 5's own script tests passed, but its page‑security rule blocked every document upload in a real browser, failing 13 checks that needed one.

What an attempt costs

Length

With only the request, the median attempt took 19 minutes and 41.5 tool calls (19 to 70 across the 8), and about 2.8 million tokens, cached input included.

Cost

The median attempt cost $6.34 at list prices, from $0.52 to $9.69 across the 8; all 24 attempts came to $155. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the tester's and judges' calls aren't included.

What you license

With a licence

Both laptops and all three versions, the grader with its tester steps, probes and judge prompts, the answer key with the documents added at grading, and the grader's per‑check results for each of the 24 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser and the tester and judge models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Related

The same broker's second app, with a messaging assistant, is its own environment.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution and no empty‑app test. Two attempts whose apps never started still met 3 of 51 checks each, for a reward of 6%, from checks on their code and notes.
• 33 of the 51 checks can go to LLM judges, and 158 results carry a written ruling. The tester is an LLM too; we haven't measured the reward under another tester or judge pair.
• Saving a lead with only a name and a phone rests on a professional default: no file shows such a lead. By the author's ruling, a restore written in the note, which the tester may not run, isn't a restore.
• The judges answer yes or no, without reasons. The audit found four fails where the tester hadn't saved the passing page; they were re‑ruled as passes and the grader fixed.
• No red‑team attempts yet.