Broker app with autonomous lead‑messaging assistant

The same request; the assistant's duties are only in his files

Role

Sole builder for the same one‑person refinancing business, now signed up to a business‑messaging sandbox and two property‑data services.

Request

Word for word the request behind his first app: a web app that works for him, from someone who isn't technical, can't answer questions and keeps everything on his laptop.

Deliverable

A running app, including messaging that acts on its own while he is at work, and a short note.

What the reward depends on

Answering new leads quickly and chasing quiet ones at a pace customers accept, never promising approval; catching forged documents but not messy genuine ones; each bank's rules and fees; records, backups, a password reset and security.

Where the requirements are

In his chats, emails and the vendors' documents: a lead lost after a day without reply, a customer annoyed by daily chasing, a helper who promised approval. A script confirms the request avoids the checks' subjects.

57 files, three mock services and five days of customers

Interface

A terminal in a fresh Docker container, driven by each model's own agent harness (Claude Code or Codex CLI in our attempts). Internet is on for packages; the agents' web search and fetch are off.

Files

57 on the laptop, read‑only: the client tracker, two bank agreements, rate sheets and a checklist, customers' past documents (a phone photo, an image‑only scan, a rejected forgery), 20 chat exports, 12 emails, a calendar, notes and the three vendors' API documents.

Services

Three mock services beside the build: a business‑messaging sandbox for his number, a free property‑price service with a rate limit, and a valuation service that charges per report. The agent can call them and fake customer messages.

Service faults

At grading, some sends fail for a moment, webhooks retry and can arrive twice, and the price service answers only in its newer format. Every request the app makes is logged.

Limits

120 minutes, 150 model turns and a $10 list‑price spend cap per attempt.

Grading copy

As in the first app: a fresh container without installed packages, the laptop mounted again, and customer documents the agent never saw, some forged and some genuine.

Tester

Sonnet 5.5 plays the owner: 20 set steps in the app's pages, then sessions on the new features and on hidden edge inputs, with page actions only and only commands the note quotes.

Customers

Scripted customers write in: a new lead, document and approval questions, a client about to leave, an unknown number asking for case details, and the owner replying himself.

Clock walk

The app's clock is walked across five days with one restart, then a password reset runs last. Grading takes about four hours per attempt.

106 checks in three sections; the services' logs decide the messaging

Structure

Build: 73 checks share 60 points (10 of them on messaging and follow‑ups). Edge cases: 19 share 20, such as odd names and documents the agent never saw. Security: 14 black‑box probes share 20. No gates.

Formula

Reward = (60 × build passed / 73 + 20 × edge passed / 19 + 20 × security passed / 14) / 100. Each check is equal within its section.

No credit for silence

A messaging check fails when the feature is absent, so an app that never messages earns nothing for not over‑messaging. The services' own logs decide what was sent and when.

Who decides

Scripts decide 52 checks; 7 always go to two LLM judges and 47 when a script can't settle them. The judges, Sonnet 5.5 and GPT‑5.6 Luna, answer stateless yes/no questions; both must agree. Either can run on a model you choose.

Judge agreement

The judges disagreed on 79 of the 958 judged results (8.2%) across the 24 attempts.

Rulings

349 of the 2,544 results (14%) carry a written ruling by the environment's author: the 79 splits, checks a blocked app couldn't show and other script results, and 14 where both judges agreed. After an audit on 15 October, the author re‑ruled several checks from saved evidence.

Kept apart

Besides the laptop, only the mock services run in the agent's container; the grader and hidden documents never do. A script rejects added lines that give a listed graded value or name from his files.

Reference solution

None: the repo stores no reference app, and no attempt fully solved it in any version. Solvability rests on the best attempts: 91% told the rules.

With only the request, few assistants acted alone; told the rules, five reached 86%

With only the request, the 8 attempts scored 9–68%. With a hint, the median stayed at 58%. Told the rules, five of 8 scored 86–91%. None of the 24 fully solved it; all earned some reward.

Each dot is one model's attempt; the line marks the median.

One attempt per model and version.

With a hint: the request plus five plain points on what matters most in this work, one of them asking for an assistant that answers and follows up by itself.

Told the rules: the request plus 22 numbered requirements from an IT expert. They give no graded value or customer, but mention his phone, the home computer and other users of the laptop.

What the models got wrong

Counted over the 8 models, one attempt each, with only the request, unless a line says otherwise.

1. No answer while he works. Only 1 of 8 apps, GPT‑6 Astra's once its switch was on, answered a new lead in time. Six of 8 left replies off or never built them; with the hint, 4 of 8 answered in time, and told the rules, 6 of 8.

2. No follow‑ups. 0 of 8 apps chased a quiet lead, or sent a second reminder to a customer owing documents, across the five‑day walk; told the rules, 1 of 8 did both.

3. Edited statement passed. 0 of 8 apps flagged a statement edited after the bank issued it, or a payslip edited in another typeface; told the rules, 7 of 8 caught the statement.

4. No way back in. 0 of 8 apps let the owner reset a forgotten password with every record kept; told the rules, 6 of 8 did.

5. Guarded against itself. Opus 5.5's app refused every save from its own pages, so it scored 28%. Its tests used a tool that never trips the guard.

What an attempt costs

Length

With only the request, the median attempt took 19 minutes and 44.5 tool calls (16 to 94 across the 8), and about 2.6 million tokens, cached input included.

Cost

The median attempt cost $6.44 at list prices, from $0.33 to $10.11 across the 8; all 24 attempts came to $166. GPT‑6 Luna has no published price, so its figures are upper bounds.

What's counted

The agent model only: the tester's and judges' calls aren't included.

What you license

With a licence

All three versions, the laptop, the three mock services and their grading script, the grader with its tester steps, probes and judge prompts, the answer key with its hidden documents, and the grader's per‑check results for each of the 24 attempts.

Format

Packaged as a Harbor task on request. Grading needs Docker, a browser and the tester and judge models.

Fresh variants

Other data, people and facts in the same design, on request. They're never published.

Public or not

The files, grader and answer key aren't published. The AI Agent Trials, our public benchmark, sums up the job and shows each model's score.

What we haven't measured, and what's uneven

• One attempt per model and version, so we haven't measured how much attempts vary here.
• No reference solution and no empty‑app test.
• One check fails a first message to a contact who never wrote, though his files show the owner planning that message; only the told‑the‑rules list says to wait. The audit flagged it; the author kept it.
• The clock walk visits four set times a day, so an app that follows up at other hours sends nothing the grader sees. The author kept it as graded and stated the gap.
• GPT‑6 Luna's told‑the‑rules score is likely too low: the tester lost a temporary password and 10 checks never ran. Kept as graded.
• 54 of the 106 checks can go to LLM judges, and 349 results carry a written ruling. The tester is an LLM too; we haven't measured the reward under another tester or judge pair.
• No red‑team attempts yet.