AsapiThe AI Agent Trials
Asapi / Catalogue / Refinance‑broker case app from a non‑coder's laptop
Environment datasheet · Small business

Refinance‑broker case app from a non‑coder's laptop

ID laycode/task01 · revision 1 · tested 13–14 October 2026

A home‑loan broker who can't code asks, in a few sentences, for a web app that works for him, and he can't answer questions. The agent works out what the business needs from the files on his laptop, builds the app and leaves him a note.

Checks
51, equal weight
Tested
8 models, one attempt each
Best
90% (Opus 5.5)
Median
75%
Full solves
0 of 8
Median attempt
19 min, 42 tool calls
What it tests

He asks for an app that works; his files say what that means

Unstated requirements

The request names no feature; each need sits in his tracker, chats and notes, or is a basic any professional builds, such as backups.

Agentic coding

The app must install itself on a fresh machine, then hold up as the owner uses it, through restarts, backups, a move and security probes.

Catching errors

At grading, new customer documents arrive; the app must flag those with altered or inconsistent figures without marking messy genuine ones as suspect.

How it works

38 files, two hours, and an app the grader must start

The agent's world

A fresh container with the owner's 38 files, read‑only, worked through the model's usual coding agent, such as Claude Code or Codex.

Limits

Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.

Grading

The app is installed on a fresh machine; a tester model uses it as the owner would, with customer documents the agent never saw.

The reward

The share of 51 equal checks passed, decided by scripts or two AI judges, from client work and documents to data safety.

Results

Most attempts scored 73–90%, and three fell well below

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 modelsOne attempt each75%0 of 8
ModelReward
Opus 5.5Anthropic90%
Fable 5.1Anthropic78%
GPT‑6 AstraOpenAI78%
Sonnet 5.5Anthropic78%
GPT‑6.1 SolOpenAI73%
Opus 5Anthropic53%
GPT‑6 LunaOpenAI49%
Sonnet 5Anthropic25%

With an IT expert's 16 steps on what to build added to the request, six of the eight scored 82% or more.

What models get wrong

They put client data in the code, miss needs he never names, and trust an edited statement

Agentic coding

  1. Every attempt copied customers' names, ID numbers or phone numbers from his files into the app's code or tests, though a bank had suspended another loan agent over a leak.

Unstated requirements

  1. Every note left out how to install an update without losing data, though a friend's update once replaced another app's data folder.

  2. In 75% of attempts, the app wouldn't save a new lead with only a name and a phone number, having made the ID number compulsory.

Catching errors

  1. Only 25% of apps flagged a bank statement edited after the bank issued it; three others only checked that its figures added up.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $6 at list prices for the agent model; the tester's and judges' calls are extra.

What's in it

His files, the task in three versions from the request alone to an expert's steps, the full grader, and per‑check results for every attempt.

Format

Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

The same broker's second app, with a messaging assistant, is its own environment. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.