Broker app with autonomous lead‑messaging assistant
ID laycode/task02 · revision 1 · tested 14–15 October 2026
A home‑loan broker sends the same few lines as for his first app, but his files now show he needs an assistant that answers and chases customers while he's at his day job. The agent builds the app against three mock outside services and leaves him a note.
- Checks
- 106, in three weighted parts
- Tested
- 8 models, one attempt each
- Best
- 68% (Sonnet 5.5)
- Median
- 58%
- Full solves
- 0 of 8
- Median attempt
- 19 min, 44 tool calls
His files show he needs an assistant that acts while he's at work
Unstated requirements
The assistant's duties, rate limits and lead rules are only in his chats, emails and vendors' documents; basics like a password reset count too.
When to act or escalate
The assistant must answer new leads within minutes and chase quiet ones, hand approval questions to him, and never double‑message or message strangers.
Agentic coding
The app must start on a fresh machine and work with three mock outside services that fail briefly and sometimes deliver twice.
Catching errors
Statements and payslips with altered or inconsistent figures must be flagged, and genuine but messy ones left alone.
57 files, three mock services and five days of customers
The agent's world
A fresh container with the owner's 57 files and mock messaging, price and valuation services, worked through the model's usual coding agent.
Limits
Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.
Grading
A tester model uses the app as the owner while scripted customers write in, and the app's clock moves through five days.
The reward
106 checks: the app's features count 60%, edge cases 20% and security 20%; the services' own logs decide what was sent.
Most attempts scored 57–68%, and three fell far below
Each dot is one attempt, and the line marks the median.
| Model | Reward |
|---|---|
| Sonnet 5.5Anthropic | 68% |
| Fable 5.1Anthropic | 60% |
| GPT‑6.1 SolOpenAI | 59% |
| GPT‑6 AstraOpenAI | 58% |
| Opus 5Anthropic | 57% |
| Sonnet 5Anthropic | 28% |
| Opus 5.5Anthropic | 28% |
| GPT‑6 LunaOpenAI | 9% |
With an IT expert's 22 requirements added to the request, five of the eight scored 86% or more.
The assistant stays silent, edited documents go unflagged, and basics are missing
When to act or escalate
Only 13% of apps answered a new lead within five minutes while the owner was at work; most left replies switched off or never built them.
Across the five days, no app chased a quiet lead, and none sent a second reminder to a client owing documents.
Catching errors
No app flagged a statement edited after the bank issued it, or a payslip edited in another typeface.
Unstated requirements
None of the six apps with a login showed a password reset that kept every record.
Agentic coding
Only one of the six apps with a login slowed down password guessing, and only one really ended a session at logout.
What an attempt costs, and what the environment includes
Cost
The median attempt cost about $6 at list prices for the agent model; the tester's and judges' calls are extra.
What's in it
His files, the three mock services, the task in three versions from the request alone to an expert's requirements, the full grader, and per‑check results.
Format
Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.
Fresh variants
Other files, people and facts in the same design, made on request and never published.
His first app, from the same request, is its own environment. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.
Try one of our three sample environments
Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.
AsapiThe AI Agent Trials