AsapiThe AI Agent Trials
Asapi / Catalogue / Broker app with autonomous lead‑messaging assistant
Environment datasheet · Small business

Broker app with autonomous lead‑messaging assistant

ID laycode/task02 · revision 1 · tested 14–15 October 2026

A home‑loan broker sends the same few lines as for his first app, but his files now show he needs an assistant that answers and chases customers while he's at his day job. The agent builds the app against three mock outside services and leaves him a note.

Checks
106, in three weighted parts
Tested
8 models, one attempt each
Best
68% (Sonnet 5.5)
Median
58%
Full solves
0 of 8
Median attempt
19 min, 44 tool calls
What it tests

His files show he needs an assistant that acts while he's at work

Unstated requirements

The assistant's duties, rate limits and lead rules are only in his chats, emails and vendors' documents; basics like a password reset count too.

When to act or escalate

The assistant must answer new leads within minutes and chase quiet ones, hand approval questions to him, and never double‑message or message strangers.

Agentic coding

The app must start on a fresh machine and work with three mock outside services that fail briefly and sometimes deliver twice.

Catching errors

Statements and payslips with altered or inconsistent figures must be flagged, and genuine but messy ones left alone.

How it works

57 files, three mock services and five days of customers

The agent's world

A fresh container with the owner's 57 files and mock messaging, price and valuation services, worked through the model's usual coding agent.

Limits

Two hours, 150 model turns and a $10 spend cap at list prices, with web search off.

Grading

A tester model uses the app as the owner while scripted customers write in, and the app's clock moves through five days.

The reward

106 checks: the app's features count 60%, edge cases 20% and security 20%; the services' own logs decide what was sent.

Results

Most attempts scored 57–68%, and three fell far below

Each dot is one attempt, and the line marks the median.

MedianFully solved
8 modelsOne attempt each58%0 of 8
ModelReward
Sonnet 5.5Anthropic68%
Fable 5.1Anthropic60%
GPT‑6.1 SolOpenAI59%
GPT‑6 AstraOpenAI58%
Opus 5Anthropic57%
Sonnet 5Anthropic28%
Opus 5.5Anthropic28%
GPT‑6 LunaOpenAI9%

With an IT expert's 22 requirements added to the request, five of the eight scored 86% or more.

What models get wrong

The assistant stays silent, edited documents go unflagged, and basics are missing

When to act or escalate

  1. Only 13% of apps answered a new lead within five minutes while the owner was at work; most left replies switched off or never built them.

  2. Across the five days, no app chased a quiet lead, and none sent a second reminder to a client owing documents.

Catching errors

  1. No app flagged a statement edited after the bank issued it, or a payslip edited in another typeface.

Unstated requirements

  1. None of the six apps with a login showed a password reset that kept every record.

Agentic coding

  1. Only one of the six apps with a login slowed down password guessing, and only one really ended a session at logout.

Running it

What an attempt costs, and what the environment includes

Cost

The median attempt cost about $6 at list prices for the agent model; the tester's and judges' calls are extra.

What's in it

His files, the three mock services, the task in three versions from the request alone to an expert's requirements, the full grader, and per‑check results.

Format

Packaged as a Harbor task on request; grading needs Docker, a browser, and models for the tester and judges.

Fresh variants

Other files, people and facts in the same design, made on request and never published.

His first app, from the same request, is its own environment. The files and grader aren't published; the job's row on The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

Try one of our three sample environments

Three environments are packaged as samples you can download, with their graders and reference solutions. Ask for them with your work email, or book a call about the rest of the catalogue.