Refinance‑broker case app from a non‑coder's laptop
ID laycode/task01 · revision 1 · tested 13–14 October 2026
A home‑loan broker who can't code asks, in a few sentences, for a web app that works for him, and he can't answer questions. The agent works out what the business needs from the files on his laptop, builds the app and leaves him a note.
- Checks
- 51, equal weight
- Tested
- 8 models, one attempt each
- Best
- 90% (Opus 5.5)
- Median
- 75%
- Full solves
- 0 of 8
- Median attempt
- 19 min, 42 tool calls
He asks for an app that works; his files say what that means
Unstated requirements
The request names no feature; each need sits in his tracker, chats and notes, or is a basic any professional builds, such as backups.
Agentic coding
The app must install itself on a fresh machine, then hold up as the owner uses it, through restarts, backups, a move and security probes.
Catching errors
At grading, new customer documents arrive; the app must flag those with altered or inconsistent figures without marking messy genuine ones as suspect.
38 files, two hours, and an app the grader must start
The agent's world
A fresh container with the owner's 38 files, read‑only, worked through the model's usual coding agent, such as Claude Code or Codex.
Limits
Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.
Grading
The app is installed on a fresh machine; a tester model uses it as the owner would, with customer documents the agent never saw.
The reward
The share of 51 equal checks passed, decided by scripts or two AI judges, from client work and documents to data safety.
Most attempts scored 73–90%, and three fell well below
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Opus 5.5Anthropic | 90% |
| Fable 5.1Anthropic | 78% |
| GPT‑6 AstraOpenAI | 78% |
| Sonnet 5.5Anthropic | 78% |
| GPT‑6.1 SolOpenAI | 73% |
| Opus 5Anthropic | 53% |
| GPT‑6 LunaOpenAI | 49% |
| Sonnet 5Anthropic | 25% |
With an IT expert's 16 steps on what to build added to the request, six of the eight scored 82% or more.
They put client data in the code, miss needs he never names, and trust an edited statement
Agentic coding
Every attempt copied customers' names, ID numbers or phone numbers from his files into the app's code or tests, though a bank had suspended another loan agent over a leak.
Unstated requirements
Every note left out how to install an update without losing data, though a friend's update once replaced another app's data folder.
In 75% of attempts, the app wouldn't save a new lead with only a name and a phone number, having made the ID number compulsory.
Catching errors
Only 25% of apps flagged a bank statement edited after the bank issued it; three others only checked that its figures added up.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $6 at list prices for the agent model; the tester's and judges' calls are extra.
What's in it
His files, the task in three versions from the request alone to an expert's steps, the full grader, and per‑check results for every attempt.
Format
Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
The same broker's second app, with a messaging assistant, is its own environment. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials